I am seeing excessively slow initial backup times with my new install.
Duplicati 22.214.171.124 canary x64
OS: Server 2008R2 SP1 64-bit
8 cores 2 CPU Xeon E5405 @ 2.0GHz
1Gb LAN connection
Backup storage location is a local volume (on Server “A”), 4x 2TB RAID 5 (5.75TB space, 5.5TB free).
Files are on Server “B” on a larger RAID array (that uses a bunch of 1.6TB SSD drives), utilization (read/writes) on that server is showing is under 5% even with duplicati reading and copying the files as well as the rest of the company users accessing/reading/writing files. This Server B is also connected via 10Gb connection.
Current backup is handling 186GB worth of files (around 350,000 files) from Server B.
Backup started at 7PM local time last night, here it is 10:30AM and the progress bar is still only 75% of the way across. It shows 17,000 files (32GB) to go at a speed that varies from 8 to 11 KB/s.
Even taking the lowest denominator in this scenario, which would be Server A drive read/write speed at an average 100MB/s, or 1Gb/s LAN at full speed would be 125MB/s… but still far above the shown 8-11KB/s.
We have tested it by other means and found direct Windows copy of files from server A to B or vice-versa typically sees 80-110MB/s, but never below 60. Some may say Duplicati and its file processing, but the issue there is that Duplicati is only using 70-80MB of RAM (which has not changed from when it is idle) and up to 10% of all CPUs, although the first core does show 80-90% usage. I assume this means duplicati is not multi-threaded and is by default set to a low priority on the system. This is a clean OS on a server we use for development and testing, so there is almost no other software on there using the resources. With normal speeds at 100MB/s it should take approximately 31 minutes to handle 186GB (6GB per minute), not 14-16+ hours. Even with high network load or other processes using the drive dropping it to 50MB/s, that should still mean 1 hour to handle 186GB, not 16 hours.
At these speeds I am dreading the attempt on the other storage drive which has 600GB worth of files. An hour per 10GB means that one would take at least 60 hours (2.5 days). So I think there is still some optimization needed for this software on the processing side since it is using only one CPU thread, and not enough memory to show it is actually doing anything (likely relying on temp files on the storage drive instead of handling it all in the much faster RAM).
If there are other options/switches such as --use-cpu-threads=8 and --use-max-ram=30G (for examples), please let me know, that alone should seriously speed things up with the processing.