I backup a few Terabytes of data using Duplicati 2! First of all, many thanks for that cool tool!
Since the backup runs rather slow I started to find the bottleneck.
The Backup-process runs with around 2.5 MB/s to 3.0 MB/s. You can imagine how long a 4TB backup runs ^^
So I benched the harddisk where the data is stored. It can read around 80MB/s.
The remote storage (same LAN, SMB) can write data with around 10MB/s
dd if=/dev/zero of=/mnt/Backup/bench bs=8k count=100k
So somewhere in the backup process I loose almost 80% of possible speed.
During a backup the multicore CPU does not seem very busy.
Are there parameters I could play around to enhance the speed of the backup?
You may be able to write 10MB/s to the remote storage using dd, but Duplicati won’t achieve that performance. This is because it is doing a lot more work behind the scenes: hashing, deduplicating, compressing data.
What advanced options have you customized?
With a 4TB backup, I recommend you change the blocksize option. The default is 100KiB but that is too small (in my opinion) for such a large backup. I would probably use 1MiB - 5MiB. Larger blocks decrease deduplication efficiency but speed up database operations as there are fewer blocks to track. Note that you cannot change this setting without starting your backup completely over.
I don’t know if this will really affect the speed of your backup though.
Increasing threads via the concurrency options may help (since you say CPU utilization is currently low) although I have to admit I haven’t experimented with those myself.
If it’s not the CPU, it’s probably the hard drive, which gets hugely slower at random I/O than at sequential.
https://hdd.userbenchmark.com has lots of actual test results showing how big a difference one can see.
iostat %iowait might be informative. If it got mechanical-drive-limited, you might need an SSD or ramdisk.
top is a good general-purpose tool that will show you a summary of
wa = I/O waiting for your system.
sar is another good tool (but CLI), and there are lots of other tools you might get to look into performance.