I have been using Duplicati for over a year now, and it “just works”. Initial uploads have however been always a pain point, and recently I decided to take a closer look.
I am on AT&T UVerse internet service. The service is advertised as 25 Mbps down, 5 up. I consistently get around 90 KBytes/s reported by Duplicati web UI. This is far lower than what is possible. So I contacted wasabi support and they had me running a bunch of tests. One of them is a speedtest to Wasabi servers. It looks like
I get somewhat better results with regular speedtest with nearby “regular” speedtest servers, but let’s go with reported 2.7 Mbps.
I then created a 63 MB test file, and uploaded to a separate test bucket using awscli to Wasabi. My router, running tomato, shows this (red line is upload)
It shows the difference in upload speed between awscli and Duplicati. awscli reports around 600 KB/sec, which is kinda freaky because it is more than the advertised bandwidth and is also more than what I got from Wasabi speedtest.
What can possibly explain the difference and how can I get Duplicati to upload faster? The machine doing the upload is a debian box with amd cpu 8 cores that is totally idle with 16GB RAM, so there is no bottleneck there. There is no throttling in Duplicati or in the router. The Duplicati job is:
Well, I have an update. I moved from debian stretch + duplicati_2.0.3.3-1_all.deb to debian buster + duplicati_2.0.4.5-1_all.deb. I have no clue why the performance was so bad (90 KB/s), but upload speed issues are gone.
Here I am pushing ~700 KB/s. That’s a more than 7x speedup. With 1.3 TB to backup, I was looking at ~180 days to back it all up. Now it is projected about ~23 days. In other words, completely impossible vs. totally doable.
Looks like the Wasabi speedtest is not very indicative. I am getting way more upload bandwidth to s3.wasabisys.com than what their bandwidth test predicted. A very welcome surprise to be sure.
Thanks for the great write up of your Wasabi test! My guess is it was the Duplicati upgrade that helped out the most on the upload side as there 2.0.4.5 beta included some new multithreading functionality.
I don’t know if Wasabi supports parallel uploads but there are some improvements in that area in the canary versions. Hopefully you’ll see even more improvement when those reach beta status!
I thought about noting that AWS CLI S3 Configuration looks like it has parallel uploads within the same file (by default, it looks like 8MB and larger files have 10 concurrent requests), but I wasn’t sure if the aws CLI advantage was just that, or if there was more going on. It turns out there was more. I’m glad you found it. Sometimes newer mono versions also help, but I hate to just toss that out as initial “I wonder if it helps?”.
Duplicati canary upload does parallel uploads of different files, but it should still help. The reason parallel uploads improve throughput is that a single network connection is limited by distance-dependent latency, however your 45 ms ping time is not that bad. Even going halfway around the world shouldn’t be so slow. Thanks for your persistence, although changing multiple things at once makes it hard to say what helped.
Yes, it was a nice write-up of the problem. Sometimes I ask for graphs in hopes of seeing some clue that the link is going through fits of slowness, then ramping back up to full speed, but yours was just smooth-and-slow. tcpdump/tshark/wireshark can sometimes get a handle on issues, but analysis is not often DIY.
