New to Duplicati, running version 22.214.171.124_canary_2018-01-23 on a QNAP TS-453A NAS on QTS 4.3.4, connecting to DreamHost DreamObjects. I have also tested this with Dropbox as well, and the same results explained below apply – so I am highly doubtful the issue is with DreamHost.
My internet is through AT&T Gigapower, with a 900+mbps download speed and an upload speed of 887mbps (Speedtest by Ookla - The Global Broadband Speed Test)
Duplicati is only uploading at a max of 10MB/second. I realize that MB and mbps are very different, however 10MB’second (around 80mbps) is not even 1/10th of my upload speed.
I have tried setting the throttle to 120MB (960mbps) and Duplicati never surpasses 10MB. In fact, it slowly drops from 10MB to 0 and then goes back to 10MB/s constantly.
How do I fix this? There absolutely has to be something wrong here. Any tips would be greatly appreciated. I am trying to upload 5+ TB of data and that’s gonna take me about 23 days at the speed it’s going.
I have pretty much the exact issue. TS-853A (same model as yours but with 8 bays) with 3.25TB of data to upload via SFTP and its going to a local client so bandwidth isn’t an issue yet Duplicati goes from 0 to 10MB/s and back down. I’m averaging around 3MB/s.
Hopefully there’s a fix for this.
How do you know that Duplicati is uploading at 10MB/s? Have you tried another way to upload data to your provided (DreamHost DreamObjects) to make sure this is not a hosting service limitation? There’s no throttling in Duplicati unless set by user, so basically it will upload as fast as the destination can download. Can you confirm that it isn’t related to the provider. Cheers.
yes, I can absolutely confirm that it is not related to DreamHost DreamObjects. I can easily upload via SFTP client (Cyberduck) at 30+MB/s, so I know it’s not them. This is specifically related to Duplicati.
I also tested with Dropbox (completely different provider) with the exact same issue. And I know it’s transferring at 10MB/s or lower because of the interface:
I turned off the throttling as well in Duplicati, with no change. And if you look in my thread, @rb28z2 is also experiencing the exact same issue.
I’m not entirely sure that the interface is showing the (correct) upload speed. I have seen it once going down every minute when it was actually not uploading anything.
Have you checked the resource monitor if CPU, disk, network or RAM might be the bottleneck?
If this system is using a dual-core CPU? If so, then it seem that one core is fully occupied. My guess is, that Duplicati is running on one core and fully occupies it with hashing. In that case, you cannot speed it up more at the moment.
It’s a quad core intel celeron 1.6Ghz processor.
yeah, might be CPU bottlenecked.
My servers quad core acts similarly. It sits at around 90% usage on two cores while backing up - even while just processing files and not requiring internet.
I guess the speed metric is interesting for predicting when the backup is done but very misleading for showing actual network usage even though it could be confused as an indication of that.
@Andrew_Kelly Also try fiddling with the size and number of dblock files to keep your upload humming. Set it to something larger like 1GB and have 10 files queued for upload if you have the space in the temp directory. It sounds like your upload speed is a lot quicker than the actual backup speed.
I think so too. You can verify it by doing a backup without encryption. You will probably see a massive increase in speed.
It’s worth mentioning that Ken is
Also I’m secretly working on LZ4. But don’t tell anyone, cause I spent more than a month on a proof of concept for the LZ4 frame format and it’s still not entirely done.
Right, of course! It’s not only encryption, it’s also compression!
Don’t worry, I can keep a secret