I’d like to backup ~500 GB of variously sized files/folders to Storj. I have a 10 MBit/s upload speed. According to the Windows task manager, Duplicati constantly uses the full 10 mbps upload.
Yet, the upload becomes slower and slower over time. At the beginning, the speed indicator of Duplicati (shown at the top of the screen) shows reasonable numbers (e.g. 1.2 MB/s), but then it drops to something like 200 KB/s. In other words: the backup process is 5 times slower than it should be. This will take weeks!!
From observing the live logs, I can tell that Duplicati starts the upload of a new 50 MB block about every 4-5 minutes (even though, mathematically, uploading 50 MB of data at full speed of 10 Mbit/s should take about 40 seconds). Something is seriously wrong here - what the hell is Duplicati using all this bandwidth for?
Here are some system details:
Windows 11
Duplicati version: 2.0.8.1_beta_2024-05-07
Source: data is stored on a local M.2 SSD (Samung EVO 990 Pro with 7+ GB/s sequential read speed)
AMD Ryzen processor - Duplicati never exceeds a few percent of CPU usage, RAM usage is normal, around 1 GB
Looking at the connections more closely, this is very concerning. What on earth is going on? Why would there be so many connections to “normal” IPs? If I didn’t know any better, I’d suspect that Duplicati just put me into a botnet.
I think I solved the problem by configuring my Storj bucket as S3 compatible destination in Duplicati. The upload is now consistently at ~0.9 MB/s and Duplicati only maintains one TCP connection (to the storj s3 endpoint), as expected.
I also set the asynchronous-concurrent-upload-limit option to 1, because parallel uploads are not really useful in this case.
Regarding the many connections: I believe that this might just be how Storj by default works. Involving end-user-machines in a huge network, for whatever reason. In any case, the upload is dead slow. You might want to put a caveat into the docs. From a performance perspective, the “Storj DCS” storage type implementation of Duplicati is completely useless… (who would willingly want uploads to happen at 1/5th of the expected speed?)
