Throttling uploads?

1

I am doing the first full backup to Onedrive of approximately 350 GB of data. With my 25 Mbps cable upload, this will take some time.

To keep room for video/audio uploads to Zoom/Teams I wanted to limit Duplicati to 8 Mbps (1 MB/s). However, if I configure the upload (and download) to be 1 MB/s, the upload still uses my complete 25 Mbps upload speed.

In Upload Throttle not working - #26 by oucil I read to set set the asynchronous-concurrent-upload-limit setting to 1. After that change, I see behaviour with peaks for 25 Mbps, then nothing, then another peak:

image

And with each peak, Zoom/Teams has issues. I expected a continuous max bandwidth of 8 Mbps (1 MB/s). I am doing something wrong? I am running:

  • APIVersion : 1
  • PasswordPlaceholder : **********
  • ServerVersion : 2.0.5.1
  • ServerVersionName : - 2.0.5.1_beta_2020-01-18
  • ServerVersionType : Beta
  • StartedBy : Tray icon
  • BaseVersionName : 2.0.4.5_beta_2018-11-28
  • DefaultUpdateChannel : Beta
  • DefaultUsageReportLevel : Information
  • ServerTime : 2021-02-02T19:26:08.7842144+01:00
  • OSType : Windows
  • DirectorySeparator : \
  • PathSeparator : ;
  • CaseSensitiveFilesystem : false
  • MonoVersion :
  • MachineName :
  • UserName :
  • NewLine :
  • CLRVersion : 4.0.30319.42000
  • CLROSInfo : {“Platform”:“Win32NT”,“ServicePack”:"",“Version”:“10.0.19041.0”,“VersionString”:“Microsoft Windows NT 10.0.19041.0”}
2 
throttle-upload

--throttle-upload = 0kb
By setting this value you can limit how much bandwidth Duplicati consumes for uploads. Setting this limit can make the backups take longer, but will make Duplicati less intrusive.

For throttling uploads, the main setting is in the throttle-upload option. In 2.0.5.1 there is a bug that also throttles downloads at that rate. It’s fixed in Canary but not yet in any Beta. And I’m not sure speeds are exactly as set but you should be able to compensate for that if you have to, by setting a different speed.

  --asynchronous-concurrent-upload-limit (Integer): The number of concurrent
    uploads allowed
    When performing asynchronous uploads, the maximum number of concurrent
    uploads allowed. Set to zero to disable the limit.
    * default value: 4

doesn’t directly set speed, though it might interact. By setting it to 1, you stop default parallel uploads, not

3

I use the toachograph setting:

image

Before starting the backup. Isn’t that the same?

4

I set the upload to 1500 kb via --throttle-upload = 1500kb but it is ignored:

image

I removed the asynchronous-concurrent-upload-limit

5

It might not be ignored so much as not give a smooth flow. This isn’t a router, which by the way can often help with traffic flows, and might do so better. Duplicati throttles by sending some data out now and then.

The problem you see might be worse on OneDrive, where the standard amount of send seems to be 10 MiB, which is going to apparently interfere with your other uses. In a test on the latest Canary, setting this

  • --fragment-size (Integer) Size of individual fragments which are uploaded separately for large files. It is recommended to be between 5-10 MiB (though a smaller value may work better on a slower or less reliable connection), and to be a multiple of 320 KiB. Default value: 10 MiB:

to 327680 in Advanced options got me smaller spikes, which maybe other applications can ride through.
This had to be done in conjunction with asynchronous-concurrent-upload-limit set to 1 in order to succeed although there’s probably a bug behind that. Parallel uploads are fairly new and might be calculating wrong.

About → System info gives a good view of the progress, and you can line its data up with network graphs.

Server State Properties section will have a lastPgEvent line that will show your BackendFileProgress that jumps every now and then. On one early test, it jumped by about 10 MiB with several minutes in between. BackendSpeed is a calculated value based on amount per time, and so ramps down after a burst is sent.

You can play with this if you like, and using the GUI speed control might be best because of the 2.0.5.1 bug where upload speed controls download speed. There may be times you want a fast restore, and the cable modem probably does downloads faster anyway. You can try running Canary, but it can sometimes break, because that’s where the new features and fixes go, and sometimes bugs are introduced along with them.

I’d also suggest seeing if your router has any speed controls that can either slow Duplicati upload down or prioritize the network activities that have real-time bandwidth needs, such as your Zoom/Teams sessions.

6

You are confirming my thoughts. I configured Windows 10 to use QoS and limit Duplicati now. Works like a charm.

1 Like
7

Tell us a little bit about how you configured wWndows for QoS.
Others might want to replicate your work.

8

I played around with:

New-NetQosPolicy -Name Duplicati -IPProtocolMatchCondition Both -AppPathName "Duplicati.GUI.TrayIcon.exe" -ThrottleRateActionBitsPerSecond 20MB -NetworkProfile All -IPDstPrefixMatchCondition 13.0.0.0/8 
Set-NetQosPolicy -Name Duplicati -IPProtocolMatchCondition Both -AppPathName "Duplicati.GUI.TrayIcon.exe" -ThrottleRateActionBitsPerSecond 20MB -NetworkProfile All -IPDstPrefixMatchCondition 13.0.0.0/8 
Get-NetQoSPolicy -PolicyStore "ActiveStore"

The -IPDstPrefixMatchCondition 13.0.0.0/8 Could be ommitted.

9

Note that if you are attempting to use QoS on a Windows PC that is not on a Windows domain, you need to set the following (and reboot):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\QoS]
"Do not use NLA"="1"
1 Like