Reduce dblock-size possible?

I have an existing backup with about 40GB and the dblock-files are 750MB.

If there a possibility to “convert” the existing backup with 750MB files to a backup with file sizes of 100MB?

Just changing the --dblock-size will only work for new files but not for the existing ones.

Background for the question: I have to change the “cloud server” and the new one has very limited download speed (limited upload asynchron dsl of the server) and so a block-size of 100MB would be better. With the existing 750MB files the verification takes a very long time. I would prefer to keep the --backup-test-samples=X, just with smaller files.

Conversion can be done locally. At the moment I have the whole backup on a local USB-storage and this storage is then tranported to the server (house of a friend).

However it should be technically possible, I can’t find a tool to do this (semi) automatically.
I was hoping that the recompress command of the Duplicati.CommandLine.RecoveryTool.exe would accept the --dblock-size option, but this option seems to be ignored.

It would indeed be a useful feature to be able to change the upload volume size.

Recovery tool does sound like the right thing to use:

Alternatively, this might do the trick (ignore the “change compresssion settings” comments) :

The recompress command in the Recovery Tool doesn’t seem to support resizing of archives, I guess the only thing it can do is switch from 7z to zip and vice versa. However, resizing arcihves would be a nice extra functionality for the recompress command.

The --small-file-size and --small-file-max-count options are a nice trick to consolidate smaller archives to a single larger one, but the OP wants to do the opposite: split a large archive to smaller archives.

Trigger rewrite of all remote volumes after –dblock-size change sounds similar. You could ask how it turned out, using –threshold=0. Whether or not that worked well, --threshold is a setting you can try to get your conversion.

Compacting files at the backend
The COMPACT command
Compact - Limited / Partial

1 Like

I tested the COMPACT process on a copy of a job and it seems to have done the trick - eventually. Feel free to test yourself, or assume my results are valid and apply to your live backup (YMMV).

My test process…

I made a copy of an existing job by:

  • copy backend files to new folder
  • export job as file
  • import job from file
  • edit name so unique
  • edit dest. to point to copy
  • disable schedule

I then ran the following against it by:

  • Go to job menu -> Commandline
  • select “Command” Compact
  • leave “Target URL >” alone
  • Remove all Commandline arguments
  • Use “Edit as text” for “Advanced options” and replace EVERYTHING with this:

It seems to have happily re-compressed 30% of the files from the default 50MB dblock size to the reqiested 40MB on the first run then stopped w/out error. I ran the process again, and it processed the rest of the files to the new 40MB dblock size.

Why it took two runs I don’t know.

I have a full-verification test running now just to make sure everything is good as this also seems to have converted my very old .7z dblock files to .zip (yay!).

Note that the progress bar just said “Running task: Starting backup …” the whole time, so estimating how long it will take is tricky. :slight_smile:

For those that care, here are the first & last few lines of the PROFILING console log:

The operation Compact has started
Starting - Running Compact
Found 0 fully deletable volume(s)
Found 0 small volumes(s) with a total size of 0 bytes
Found 93 volume(s) with a total of 0.00% wasted space (0 bytes of 4.08 GB)
Compacting because there is 0.00% wasted space and the limit is 0%
Starting - RemoteOperationList
Backend event: List - Started:  ()
  Listing remote folder ...
Backend event: List - Completed:  (276 bytes)
RemoteOperationList took 0:00:00:00.035
Starting - CommitCompact
CommitCompact took 0:00:00:00.262
Running Compact took 0:00:28:07.987
Return code: 0

Nice hack of the compact command!
I guess is has never been the intention to use this command for resizing remote volumes, but it looks like this is more convenient to use than downloading everything to local storage, converting and re-uploading with the Recovery tool.

This statement did the job for me. I didn’t need 2 executions like JonMikelV? First rund did the job!

Thanks a lot.

"c:\Program Files\Duplicati 2\Duplicati.CommandLine.exe"

Glad it worked for you! :smiley:

Do you think this was a simple enough process to be considered a “solution”?

Because I’m a glutton for punishment, I wiped my first test and repeated it - this time it handled everything in a single run, so I’m guessing something interrupted my first test.