Hi, I’m running Version 2.0.5.1_beta_2020-01-18. I have it running in a docker container since about three weeks and it worked fine for the first two. But since I updated to the new Version my backupjobs are uploading faster than I allow them to. I set all upload throttles that I could find to 1500 kbyte/s, but i mostly see speeds between 1.7 and 2 MB/s. The speed varies a lot and regularly maxes out my 28 Mbit of upstream. I set the throttle next to the progressbar, in each backupjob and in the settings menu, but none of them seem to have any effect. Am I missing something?
Welcome to the forum @Cathnan
Try setting the download too. Sometimes they’re confused, and confusion hasn’t been sorted out yet…
Hi, thank you. I tried this now, but it doesn’t work either. It’s the exact same behaviour. Is there anything else I can try?
What Storage Type is this on the Destination screen?
Was this 2.0.4.5/23 before 2.0.5.1 (which hasn’t been out for 3 weeks yet), i.e. for the two fine weeks?
One difference is parallel uploads. You could try cutting that back to 1 concurrent on screen 5 Options
I’m uploading to Jottacloud. I’m not sure what the previous Version was. It was a Beta Release from 2018. Presumbaly 2.0.4.3 or 2.0.4.5. I set the parallel uploads to 1. I’m currently seeing a more steady speed. But I do still have spikes that max out my connection and the steady speed is still at 1.64MB. So it’s still faster than the limit I set.
1 Like
1500 KB is 1.5 MB. 1.64 MB is very close to what you set.
I just tried Google Drive at 100 KBytes/sec up and down, and saw peak uploads of 1.1 Mbit/sec, which is a little higher than the 800 Kbit/sec that the math might call for, but I think it’s an average, after gaps.
I think below speed from –log-file-log-level=profiling is how a requested 100 KByte/sec turned out:
2020-01-24 18:42:33 -05 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.86 MB in 00:01:44.5579796, 96.54 KB/s
You might get spikes if the network gets slow for awhile, then there can be a catch-up race to average. You might be able to force this by pulling a network cable for a bit, but not so long as to fail the upload.
Duplicati does not have network-level control of the sending speed. If you need that, your router might.
I don’t know why anything changed since your prior version, but there were definitely changes such as parallel uploads that possibly modified some behaviors. I don’t have Jottacloud, so I can’t test with that.
Same problem for me,
Duplicati in docker container on Unraid, backing up to OneDrive and the throttling won’t work and I can not stop/pause the transfer in UI, taking all my bandwidth on network.
Solution that seems to work for me is as “ts678” mentioned, setting both upload and download throttling as well as setting “asynchronous-concurrent-upload-limit” to 1.
Kru-x
1 Like
My Google Drive test left --asynchronous-concurrent-upload-limit at default 4. I don’t know the exact algorithm (and have not tested at all settings), but a good plan might be to split the allowed throttling across however many concurrent uploads are allowed. Concurrent uploads are more for people who desire more bandwidth used. Network latency limits how much can be pushed over one connection.
So I guess we can say that a flat “not working” isn’t the case, but I’m not sure what exact issues exist.
If anyone characterizes it well enough, the best place to get it in the line for a fix someday is in Issues.
Jumped the gun too soon, suddenly Duplicati is using all my bandwith and I have to shut down the container to get it back again, still says download limit to the set value and I can not pause it or stop it from the UI.
@kru-x You may want to update the container. I’m running the linuxserver.io container on Unraid and updated about 3 hours ago. Since then I have quite steady speeds between 1.44 and 1.47 MB/s which is what I expected. I’m not sure if the container update is the reason because, I forgot to restart the upload after I changed the parallel uploads to 1, but at the moment it works as expected since a few hours ago. I’ll have to see if it stays like this.
If there’s a container-related issue, I’m not set up to look into those. It’d be kind of puzzling, because I think throttling is done by Duplicati writing data into (or reading from) OS’ networking at specified rate. Beyond that, it’s up to the OS, the router, etc. to do low-level data moves. Do containers change that?
I did think of other possible causes of spikes, although I suspect the main goal was average speed of upload and download, possibly not including authentication delay, and maybe not accounting for TCP slow start. Some short operations such as authentication and directory listings may also be full speed.
Doing a thorough analysis would probably need watching About --> Show log --> Live --> Profiling and the network speed at the same time, trying to look for spikes, guessing at cause, and maybe changing.
There are a few Storage Destination types that can’t throttle, but JottaCloud should as it’s one of these:
Good luck on the tests, welcome to the forum @kru-x, and please file an Issue if bug gets more solid.
From what I know about containers I think they can be made in a way so that they can modify data Streams at quite a low level. But I don’t think there would be a reason the create this container in such a way. The restart of the download probably helped more than the container update. I’ve seen consistent speeds all day. I do still have spikes, but they are smaller, less than a second and way less common. So now it is working as intended. Thank you for your help 
1 Like
I have this issue with Jottacloud too. Nothing will stop it using all available bandwidth.
Edit: Okay, it works if:
- Stop upload
- Set throttle
- Set threads to 1
- Restart upload
So you can’t change it during an upload. Also stopping the upload completely broke the backup, I had to delete and recreate it.
What Duplicati version? Before 2.0.5.1, there was an issue opened on throttle direction confusion:
--throttle-upload affects restore download performance #3272
2.0.5.1 seems to confuse them differently. If symmetric speeds are OK, set upload AND download.
Regarding stopping, which variety did you use? “Stop now” is a harder stop than the slower option.
Safest stop is likely 2.0.5.1 “Stop after current file”, which also has a changed name from previous.
Stop now results in “Detected non-empty blocksets with no associated blocks!” #4037 is fixed, but wasn’t fixed in time for 2.0.5.1 Beta. I think Canary is fixed, but Canary has new code and is a risk.
2.0.5.1_beta_2020-01-18
Stop now does eventually seem to work. I think the UI gets out of sync. Like the complete % is random, it goes up and down all the time too. Thanks for the info on the other issue.
“Stop after current file” is definitely slow, because there’s a long pipeline of work-in-progress to finish.
“Stop now” is something I avoid because it’s sometimes a source of problems – but is getting better…
Throttling is still rather mysterious. My tests on 2.0.5.1 are showing upload throttle controlling both the upload and download rate, and download throttle controlling nothing, which is a change from before…
Maybe there’s also something specific to Jottacloud (which I don’t have) that makes throttling worse…
I hope to have some pull requests up soon (maybe by the end of the week) to drastically make this better, or cross my fingers, to have it work properly.
1 Like
I’m running into the same thing. Here are the specifics:
Running on Linux Mint latest version, up to date.
Duplicati - 2.0.5.103_canary_2020-02-18
Backing up to Google Drive
asynchronous-concurrent-upload-limit 1
Upload and Download limits both set to 1 KByte/s
Current upload speed 209 KB/s
Pause does not work
Stop running backup does not work
It’s currently backing up a new 6 GB file
My internet connection is virtually unusable due to upload being completely saturated
I hope someone can figure out what’s going on here.
Try setting the limit before you start the backup. That works for me.
–throttle-download ignored, --throttle-upload throttles download too #4115
is the issue I opened. YMMV but behavior seems to have changed in 2.0.4.16 Canary (whose code is newer than 2.0.4.23 Beta because 2.0.4.23 Beta was 2.0.4.5 Beta plus a warning that had to get out).
There’s a table there with results. If on a Beta before 2.0.5.1, try throttling download to throttle upload, however 2.0.5.1 has so many reliability fixes that I’d suggest 2.0.5.1 and setting upload throttle even if download also gets throttled. If you need to do a big restore, maybe just turn off all throttling and run…