Is it possible to change the compression module later on or only when creating a new backup task?
What did you have in mind to change it to? I’m not sure there’s another that’s recommended.
--compression-module (String): Select what module to use for compression
Duplicati supports pluggable compression modules. Use this option to
select a module to use for compression. This is only applied when
creating new volumes, when reading an existing file, the filename is used
to select the compression module.
* default value: zip
The Factsheet says:
Duplicati supports Zip/Deflate or 7z/LZMA2 compression.
I’d expect better compression from 7z and would like to change it but I don’t want to redo the backup.
Perhaps, but .7z archive format raises some issues described here by the original Duplicati developer.
What would be more reasonable would be to stick with .zip archive format, and change compression:
--zip-compression-method (Enumeration): Sets the Zip compression method
This option can be used to set an alternative compressor method, such as
LZMA. Note that using another value than Deflate will cause the
zip-compression-level option to be ignored.
* values: None, Deflate, BZip2, LZMA, PPMd
* default value: Deflate
I would note that LZMA2 format uses a mixture of LZMA compression and no compression. Duplicati’s handling of incompressible files is to recognize those files by their file extensions. List is changeable in compression-extension-file. For better compression, you can also raise blocksize from default 100 KB.
Each block is stored as a (maybe) compressed file in the .zip file. Compressors might like larger files.
The options are limited. Over time, I would guess that compacting would recompress some dblock files.
2 Likes
But it would not be a problem for restore to have mixed compression or blocksize? That was my biggest fear.
The deduplication blocksize cannot be changed on an existing backup (where some backend data already exists), but the zip compression method can.
1 Like
Compression methods are given in the .zip file format, and set the decompression automatically if the program being used supports that method. Windows File Explorer seems to do only None and Deflate. This won’t bother you unless you open Duplicati backup files directly. If so, get a better program such as 7-zip which appears to support all the compression methods that Duplicati can do, and it can show you:
PPMd made the smallest block in my one little test of files just over 102400 (default block size), but it’s possible that the data (all files were Duplicati .dll files) just landed that way. If you want to benchmark extensively, feel free. I can’t recall (and can’t find) that anyone else has.reported results from changing.
Regardless, my test used all five options, a different one per file (files added over five backups), and the restore (with no-local-blocks set to force five dblock downloads to get source file block) worked just fine.
Some random compression tests. Source’s my Skyrim and Fallout 4 saves folders. Default Duplicati settings.
Skyrim (1,95 GiB):
7z compression format - size: 548 MiB - time 11m
Deflate (max) zip - size: 928 MiB - time: 4m
Zip with Lzma compression - size: 882 MiB - time: 11m
Fallout (3,64 GiB):
7z compression format - size: 872 MiB - time: 19m
Deflate (max) zip - size: 1.79 MiB - time: 8m
Zip with Lzma compression - size: 1.6 MiB - time: 22m
Lzma generally is a much slower method without obviously better compression results. 7z is another story, but there are some problems and odd behaviors (as previously noted). As I understand 7z is not recommended because is not stable (yet?).
2 Likes
Thanks for all the feedback. It seems it is not really worth it.
There are definitely tradeoffs, and decompression speed also factors in, and hasn’t been examined…
FWIW which isn’t much given I was mainly testing your compatibility worry, not compression metrics:
102 Original default block in KB. Each compression method used different file
102 None
45 Deflate
43 BZip2
45 LZMA
39 PPMd
Modern Lossless Compression Techniques: Review, Comparison and Analysis
gets into this quite deeply, but can’t account for your particular data, or specific code that Duplicati uses.