Backup slows download speeds

ttfn · September 10, 2017, 8:03am

Hi

Doing my first full backup (OSX 10.12.6) to Onedrive. The backup is slow, but my fibre upload speed is only 4Mbs. However, the backload seems to hammer my upload speeds (normally 40Mbs but can go down to a few Mbs), which improve if I pause the backup.

Activity monitor shows low levels of inbound data but lots of packets.

Any ideas - is this a bug or expected performance.

Tx

JonMikelV · September 10, 2017, 6:06pm

ttfn, did you use any custom settings such as for --blocksize or --dblock-size?

ttfn · September 10, 2017, 6:18pm

no - I used all the defaults

drakar2007 · September 10, 2017, 10:10pm

Do you have asymmetric upload/download from your ISP? Back in my days on DSL, any sort of uploading (torrenting, etc) would absolutely destroy my download speeds. I’d be surprised if that was an issue on fiber, but I’m not sure what some providers do versus other providers.

JonMikelV · September 11, 2017, 1:52am

ttfn, it sounds like this is something a bit unique to your setup as I’m not hearing of anybody else being able to replicate the issue. As that’s the case, perhaps some general investigation is in order.

For example, on Windows I can look at a task manager (or certain firewall tools) that tell me exactly how much bandwidth is used by what apps / services. Is there a similar thing available for OSX 10.12.6 that might help to at least confirm bandwidth is the issue? I ask because it’s POSSIBLE that something else is the constraining factor such as disk IO or CPU load that is only being ‘exposed’ by it’s effect on transfer speeds.

For example, if you have antivirus running at the packet level but your CPU is so busy compressing archives locally that can’t get to the packets in real time it would appear that your bandwidth is being fully consumed. Actually, you could probably test that by setting the --thread-priority parameter to low (or idle) and see if that has any effect.

ttfn · September 11, 2017, 7:51pm

OK, so it looks like darker2007 is right and it is a general issue with my connection, it is fibre but very asymmetric speeds (up 40Mbps down 4 Mbps) and I can recreate the effect just by copying a bunch of files into my dropbox folder and monitor the download speed whilst it is synching

I found this a useful explanation:

so now the question is can I restrict the upload speed within Duplicati so that it doesn’t create this undesired effect, or do I have to do this through my router - but the traffic management & diffserve settings are obscure

Thanks for the helpful comments

JonMikelV · September 11, 2017, 8:09pm

ttfn, sure you can limit bandwidth - just edit your backup set, go to " step 5" (options) then use the “Add advanced option” selector to choose throttle-upload (and throttle-download if applicable).

If you just want to test various speeds, you can also use the “speedometer” icon in the UI (to the right of Play / Pause) to try different settings for the currently running job to see what effect they have on your total bandwidth. Note that the speedometer are for the current run only and will not be saved with the backup set options.

ttfn · September 13, 2017, 8:50pm

JonMikeIV

that has proved helpful thanks and now I have throttled backups running on 2 machines and leaving enough bandwidth for other activities. reading some of the other posts it looks like others are wanting to run backups at scheduled times to avoid similar bottlenecks and utilising cron and the pause/resume script looks to be one option. But would it be possible to adapt the script to throttle/unthrottle the upload which would give even greater flexibility. In other words is this available through the API and how would one do this in a python script.

Other that that, I’m really please to have discovered Duplicati as an alternative to CP home. for my needs it is better and it is great to have the support of the community

JonMikelV · September 13, 2017, 8:54pm

ttfn, if you are asking if the throttling amount can be scheduled the answer is not at the moment. I suspect it’s techinically possible but the code to do so (and support it in the UI) doesn’t exist. Yet. If you’d like to change that, feel free to let us know.

Beyond that, if you feel this Topic is resolved it would be great if you could click the Checkmark box (and maybe the as well) on the Topic that resolved it for you!

kenkendk · September 14, 2017, 8:41am

Yes, it is possible to do it with the server API.
You can adapt this script:

github.com

duplicati/duplicati/blob/master/Tools/Commandline/pause-resume.py

#!/usr/bin/env python3

# Example pause/resume script
# written by will@sowerbutts.com

import requests
import urllib.parse
import sys


class DuplicatiServer(object):
    def __init__(self, base_url):
        self.base_url = base_url
        self.cookiejar = requests.cookies.RequestsCookieJar()
        self.headers = dict()

    def fetch(self, path, post=False, data=None):
        for attempt in range(2):  # we may get a "Missing XSRF token" error on the first attempt
            if post:
                r = requests.post(self.base_url + path, cookies=self.cookiejar, headers=self.headers, data=data)

This file has been truncated. show original

And then add something like:

def throttle(self, up, down):
    return self.patch('api/v1/serversettings',  json.dumps({'max-download-speed': down, 'max-upload-speed': up}))