Compact doesn't seem to do anythikng

Support
1

Hi. I am trying to compact a remote backup on Jottacloud, with access via rclone.

When I run the compact command it finishes in a few seconds with return code 0, and doesn’t remove any of the hundreds of small archives. I tried adding the following to the command:

--small-file-max-count=0
--small-file-size=0b

But that did not help. For completeness, here is the advanced options (started via the web interface):

--small-file-max-count=0
--small-file-size=0b
--backup-name=<redacted>
--dbpath=<redacted>
--encryption-module=aes
--compression-module=zip
--dblock-size=500MB
--passphrase=<redacted>
--retention-policy=1W:1D,4W:1W,12M:1M
--blocksize=10MB
--disable-module=console-password-input

Am I doing something wrong? I ran the same on another backup and it did remove small archives, bundling them together into new ones. EDIT: I take that back, it doesn’t seem to have worked properly on the other backup either, although it did do some compacting.

2

I spotted an error. The code I copied sets the minimum size to 0b, i.e. all files are above it. Set it to 45MB and it did compact one archive, but looking on the cloud server I can see that there are still lots of dblock.zip.aes files that are smaller than 45MB and untouched.

3

I haven’t used either of those switches but in reading the manual for small-file-size it is a percentage of the dblock file that is wasted, once exceeded it triggers a compact on that dblock. I’m not sure if setting it to 0 will force it to try every single time (wasting time and resources) or if it disables it. The default of 20 seems pretty reasonable and even down to around 5 but lower than that seems excessive.

Regarding using small-file-max-count, the manual for it says the following “The small volumes will always be combined when they can fill an entire volume.” So until you have enough small dblocks to fill a whole dblock (500MB in your case) they won’t get combined.

1 Like
4

Thanks. The small-file-max-count description reads

To avoid filling the remote storage with small files, this value can force grouping small files. The small volumes will always be combined when they can fill an entire volume.

I interpreted that as if I set the value to zero it will force all files below the threshold to be grouped, but apparently not. If/when I have time I should look at the Duplicati source code to try to determine exactly how this value is used.

5

Before looking at code, you could look in a verbose log-file to see what these messages are claiming:

6

Okay so the actual logic is

m_smallvolumes = from n in m_report where n.CompressedSize <= smallfilesize && !m_cleandelete.Contains(n) select n;

LocalDeleteDatabase.cs line 234.

So if the file isn’t marked for deletion anyway and is smaller than smallfilesize, which I verified comes from small-file-max-count if supplied, it gets compacted.

Line 261 of the same file has the logic to report when it is happening, but I’m not seeing the message it should produce.

Line 279 is the logic that decides if compacting will take place. It looks correct.

I can’t see anything wrong here. Also, I’ve had some success with it now. Maybe there was an issue with rclone or Jottacloud where I store the data, but testing it again I have now managed to compact about 1.6GB of small files into larger one with a different backup set. I will keep experimenting and try to get the one that failed to work.