Compact without download volumes

ts678 · February 2, 2022, 6:59pm

might be harder if you

If you leave it enabled and set up logs at whatever verbosity level you can stand, you get messages like

duplicati/duplicati/blob/27ff8cd20b3e349f6efd7b3b754637c0d940ea9c/Duplicati/Library/Main/Database/LocalDeleteDatabase.cs#L248-L265


      
          public void ReportCompactData()

          {

              var wastepercentage = ((m_wastedspace / (float)m_fullsize) * 100);

              Logging.Log.WriteVerboseMessage(LOGTAG, "FullyDeletableCount", "Found {0} fully deletable volume(s)", m_deletablevolumes);

              Logging.Log.WriteVerboseMessage(LOGTAG, "SmallVolumeCount", "Found {0} small volumes(s) with a total size of {1}", m_smallvolumes.Count(), Library.Utility.Utility.FormatSizeString(m_smallspace));

              Logging.Log.WriteVerboseMessage(LOGTAG, "WastedSpaceVolumes", "Found {0} volume(s) with a total of {1:F2}% wasted space ({2} of {3})", m_wastevolumes.Count(), wastepercentage, Library.Utility.Utility.FormatSizeString(m_wastedspace), Library.Utility.Utility.FormatSizeString(m_fullsize));

          

              if (m_deletablevolumes > 0)

                  Logging.Log.WriteInformationMessage(LOGTAG, "CompactReason", "Compacting because there are {0} fully deletable volume(s)", m_deletablevolumes);

              else if (wastepercentage >= m_wastethreshold && m_wastevolumes.Count() >= 2)

                  Logging.Log.WriteInformationMessage(LOGTAG, "CompactReason", "Compacting because there is {0:F2}% wasted space and the limit is {1}%", wastepercentage, m_wastethreshold);

              else if (m_smallspace > m_volsize)

                  Logging.Log.WriteInformationMessage(LOGTAG, "CompactReason", "Compacting because there are {0} in small volumes and the volume size is {1}", Library.Utility.Utility.FormatSizeString(m_smallspace), Library.Utility.Utility.FormatSizeString(m_volsize));

              else if (m_smallvolumecount > m_maxsmallfilecount)

                  Logging.Log.WriteInformationMessage(LOGTAG, "CompactReason", "Compacting because there are {0} small volumes and the maximum is {1}", m_smallvolumecount, m_maxsmallfilecount);

              else

                  Logging.Log.WriteInformationMessage(LOGTAG, "CompactReason", "Compacting not required");

          }

so I think you can at least sometimes go straight for delete if it gets to completely empty before threshold.

--threshold=<percent_value>
The amount of old data that a dblock file can contain before it is considered to be replaced.

2022-01-21 15:54:28 -05 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-FullyDeletableCount]: Found 2 fully deletable volume(s)
2022-01-21 15:54:28 -05 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-SmallVolumeCount]: Found 2 small volumes(s) with a total size of 8.82 MB
2022-01-21 15:54:28 -05 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-WastedSpaceVolumes]: Found 24 volume(s) with a total of 6.03% wasted space (824.01 MB of 13.35 GB)
2022-01-21 15:54:28 -05 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 2 fully deletable volume(s)

Setting compact threshold to 0% will maybe compact eternally. I haven’t tested setting it to 100% though.

This probably merits another look, but trying to optimize it for different storage pricing is kind of complex.
Threshold too low makes churn and eats bandwidth Setting high maybe holds wasted space too long…

Compact - Limited / Partial is a big discussion on this, but ultimately any changes may need a volunteer.
Duplicati’s progress and success is very much limited by people volunteering to contribute in some way.