Deleting files - excessive download

ts678 · January 23, 2023, 4:18pm

Are you certain about that? After seeing the results of a threshold change from my EDIT 3,
it’s looking like maybe it looks at volumes with too much wasted data then adds up wastes.
Comparison to threshold on total backup then follows, but (sadly) thresholds are the same.

https://duplicati.com/assets/Block-basedstorageformat.pdf which was cited yesterday says:

The initial version is to calculate the size of blocks that cannot be reached in each volume, and
once a certain threshold is exceeded, the volume is marked as wasteful. As soon as the overall
waste exceeds the size of a volume, wasteful volume(s) are downloaded and new volumes are
written.

Aside from “overall waste exceeds the size of a volume” (fixed size replaced by threshold option?), paragraph processing order fits my finding that raising threshold reduces quantity of volumes listed (“marked as wasteful” concept above), and of course summing fewer volumes reduces total waste.

github.com

duplicati/duplicati/blob/666b2281032460254839fdc3b6e1055fdf7ce1db/Duplicati/Library/Main/Database/LocalDeleteDatabase.cs#L228-L246


      
          public CompactReport(long volsize, long wastethreshold, long smallfilesize, long maxsmallfilecount, IEnumerable<VolumeUsage> report)

          {

              m_report = report;

              

              m_cleandelete = (from n in m_report where n.DataSize <= n.WastedSize select n).ToArray();

              m_wastevolumes = from n in m_report where ((((n.WastedSize / (float)n.DataSize) * 100) >= wastethreshold) || (((n.WastedSize / (float)volsize) * 100) >= wastethreshold)) && !m_cleandelete.Contains(n) select n;

              m_smallvolumes = from n in m_report where n.CompressedSize <= smallfilesize && !m_cleandelete.Contains(n) select n;

          

              m_wastethreshold = wastethreshold;

              m_volsize = volsize;

              m_maxsmallfilecount = maxsmallfilecount;

          

              m_deletablevolumes = m_cleandelete.Count();

              m_fullsize = report.Select(x => x.DataSize).Sum();

              

              m_wastedspace = m_wastevolumes.Select(x => x.WastedSize).Sum();

              m_smallspace = m_smallvolumes.Select(x => x.CompressedSize).Sum();

              m_smallvolumecount = m_smallvolumes.Count();

          }

looks to me (who doesn’t do much C#) like what I described, but it’s just the report, not the compact.
Assuming all is well, the key point is that it’s not looking at the total waste, just the waste in volumes individually exceeding the waste threshold, so if we can ever get the dual-threshold you proposed, a possible solution would be to set per-volume threshold at some generous amount, set total threshold based on how often we’d hope to compact. Compacting requires several volumes to be available, as created new volumes get filled, and by definition a volume with waste won’t fully fill one all on its own.

Does it just upload a small file then? You can see that most of mine were 5% full or less, and very little would be uploaded back. This possibly unusual pattern would better suit Duplicati’s pack-it-full method. Even restic seems like it could have a flurry of compacts if a lot of volumes hit waste limit all at once…

I think the dual threshold might be easier, and maybe enough to have more but shorter compact runs. Regardless, nothing (in any code area) happens without developers, and any interested one can look.