I’m using a Raspberry Pi 4 with 8 gb of ram as a NAS at home, encrypting the shared USB3-connected drive using dm-crypt/LUKS with the xchacha12,aes-adiantum-plain64 algorithm.
Taking a backup using my desktop computer to Google Drive pretty much maxes out at full speed, limited only by my broadband (500 mbit). The RPi4 however is much slower, and only reaches approximately 4,5 MB/s (around 40 mbit).
My drive/rpi is capable of higher speeds, as uploading files in any other way (scp, ftp, nfs etc) maxes out at around 75 MB/s (600 mbit).
Is there any option or combination of options I can use to give the upload speed a boost? 6,5 gb of ram is used for a shared buffer/cache, and the entire system uses less than 0,5 gb of ram (including Duplicati). Can I make Duplicati somehow use more memory to speed it up, as the OS buffering doesn’t seem to make much difference? How do I turn off compression or tweak the compression level to see if it changes anything?
I’ve tried the following options in various combinations:
blocksize and dblock-size: 1 mbyte up to 100 mbyte
http-enable-buffering: true
use-block-cache: true
asynchronous-concurrent-upload-limit: 1 up to 4
asynchronous-upload-limit: 1 up to 4
None of them, however, seem to have any impact at all on the upload speed.
Have you made it past the first initial backup? That 4.5MB/s seems pretty common during the initial backup even on hardware that can move much more data per second. Once the first backup is made things speed up massively but that first one is painful. I’m not sure what’s holding Duplicati back from using more resources but something is.
I would always suggest using the -use-block-cache option but generally most other options are probably best left at defaults, particularly the block-size.
Duplicati.CommandLine.exe help zip
Zip compression (.zip):
This module provides the industry standard Zip compression. Files created
with this module can be read by any standard-compliant zip application.
Supported options:
--zip-compression-level (Enumeration): Sets the Zip compression level
This option controls the compression level used. A setting of zero gives
no compression, and a setting of 9 gives maximum compression.
* values: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
* default value: BestCompression
--compression-level (Enumeration): Sets the Zip compression level
[DEPRECATED]: Please use the zip-compression-level option instead
This option controls the compression level used. A setting of zero gives
no compression, and a setting of 9 gives maximum compression.
* values: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
* default value: BestCompression
--zip-compression-method (Enumeration): Sets the Zip compression method
This option can be used to set an alternative compressor method, such as
LZMA. Note that using another value than Deflate will cause the
zip-compression-level option to be ignored.
* values: None, Deflate, BZip2, LZMA, PPMd
* default value: Deflate
--zip-compression-zip64 (Boolean): Toggles Zip64 support
The zip64 format is required for files larger than 4GiB, use this flag to
toggle it
* default value: False
Duplicati uses the drive more. In addition to reading, it makes temporary files destined to become .zip archives eventually, and it encrypts them (I didn’t hear that you don’t want that – but it can be disabled).
It’s also updating the job’s local database constantly, and all of this is probably slow random accesses.
For any performance tuning, actual measurements are always helpful, for example from sar or similar.
Does your desktop use an SSD? Some people on the forum would prefer Duplicati write less to theirs.
RAM disk is nice and fast, and maybe less prone to write wear, but I’m not sure the Pi has the memory.
I ‘solved’ this by using a workaround letting my desktop computer run duplicati and chug the files from my RPi NAS using NFS, instead of running duplicati on the RPi itself.
Whilst I’m unable to max out at 75 MB/s, atleast this way it now uploads at roughly 50 MB/s which is more than 10 times faster than before.
