Why does upload speed fluctuate

Support
1

I am uploading a test backup to pcloud and since I had my Windows Task Manager open, I noticed that noticed that sometimes the upload speed is around 3 Mbit/s and sometimes it’s around 7 Mbit/s:

image
image.png640×593 28.3 KB

The screenshot shows the point where it just changed from 3 to 7 (or even a bit more, my official upload bandwidth is 7-10 Mbit/s)

The fluctuating upload speed are confirmed by the log (I added arrows to indicate the points where upload was slower than normal):

6 Dec 2017 23:06: put duplicati-if9b9be4b73cd4af9ac66243fd00e9ddf.dindex.zip.aes
->
6 Dec 2017 23:02: put duplicati-bbdad02aa72e34167a424cd7faa4b73de.dblock.zip.aes
6 Dec 2017 23:02: put duplicati-id6afd22d354546b298e3b44ff9134432.dindex.zip.aes
6 Dec 2017 23:01: put duplicati-b2991a82d96764f6db53a734a131edafd.dblock.zip.aes
6 Dec 2017 23:01: put duplicati-i25d84aedf0a84b2cad85cdb6e648f99e.dindex.zip.aes
6 Dec 2017 23:00: put duplicati-be2b64e4c60164538b2764dfb58f94987.dblock.zip.aes
6 Dec 2017 23:00: put duplicati-ib984699687144e89aca3a4b5665e5c36.dindex.zip.aes
6 Dec 2017 23:00: put duplicati-bbb354fa0f7d8442ab61be4c8446ea786.dblock.zip.aes
6 Dec 2017 23:00: put duplicati-i354c2cac48ec4f2e9882668f53d3fcae.dindex.zip.aes
->
6 Dec 2017 22:55: put duplicati-bdb80dade1b36436aa56cbae2ada15647.dblock.zip.aes
6 Dec 2017 22:55: put duplicati-i843810073d174e349ab9db4ff662d883.dindex.zip.aes
6 Dec 2017 22:55: put duplicati-ba85c5634af19497990cf59f3aef38fc0.dblock.zip.aes
6 Dec 2017 22:55: put duplicati-i88b41d08230647b18faf643aac11c15a.dindex.zip.aes
6 Dec 2017 22:54: put duplicati-i72323d96064d448189866f2e7c2f8fba.dindex.zip.aes
6 Dec 2017 22:53: put duplicati-b59225d9a300940a68c4d3a37eafc5f03.dblock.zip.aes
6 Dec 2017 22:53: put duplicati-i41f7bb906f0a450eba7636c93936c4cf.dindex.zip.aes
->
6 Dec 2017 22:48: put duplicati-b65ee652ea4224b44934326d6e0814ac2.dblock.zip.aes
6 Dec 2017 22:48: put duplicati-ia268213fa99c4fd89aaf91f028d766d7.dindex.zip.aes
->
6 Dec 2017 22:44: put duplicati-bd5fdb66f34dc49fc9a7ee0ef3b0b63d2.dblock.zip.aes
6 Dec 2017 22:44: put duplicati-i279159b340ed434093e82d7836e90559.dindex.zip.aes
6 Dec 2017 22:43: put duplicati-b4864404dbbba4f8d8e9331a5b4bb8034.dblock.zip.aes
6 Dec 2017 22:43: put duplicati-i847920daf5fa4299a0af224235295698.dindex.zip.aes
6 Dec 2017 22:42: put duplicati-bdc560d2e8ef047c0b9d550563099e17d.dblock.zip.aes
6 Dec 2017 22:42: put duplicati-i8507f061873148b5a8678c34b0dd79d9.dindex.zip.aes
6 Dec 2017 22:41: put duplicati-ba5da24b1f9e74331b6f0e085fb900ae8.dblock.zip.aes
6 Dec 2017 22:41: put duplicati-i89f66723789c49cc92e1fa46bfa48ed3.dindex.zip.aes
6 Dec 2017 22:40: put duplicati-b1418b3696b4d4c87b24a9181ec8fba56.dblock.zip.aes
6 Dec 2017 22:40: put duplicati-if3d07d5583da459f82674aec4369daf5.dindex.zip.aes
6 Dec 2017 22:39: put duplicati-b43954c8fda0d4f43b0691eeb55bf8ddb.dblock.zip.aes
6 Dec 2017 22:39: put duplicati-i5ebd967c14954331a3c03976cbd52b65.dindex.zip.aes
6 Dec 2017 22:38: put duplicati-b064f7c372491455d9ec5525447f15686.dblock.zip.aes
6 Dec 2017 22:38: put duplicati-i03c87ecdcda94282a6d817f4c2dc83ef.dindex.zip.aes
6 Dec 2017 22:36: put duplicati-b709be96c941048c78064500680c833ae.dblock.zip.aes
6 Dec 2017 22:36: put duplicati-i1ae5f7647ccc41599b3c53e6087fa341.dindex.zip.aes
->
6 Dec 2017 22:31: put duplicati-bae6b64d571584f21855c7f1355948f46.dblock.zip.aes
6 Dec 2017 22:31: put duplicati-i40c94937511d49ae825241a3b85f890a.dindex.zip.aes
6 Dec 2017 22:30: put duplicati-b0d8cc8a3c4c4479a97370bfb0d46ffe6.dblock.zip.aes
6 Dec 2017 22:30: put duplicati-i14fc671a7fa64c4d88537598081f0f99.dindex.zip.aes
->
6 Dec 2017 22:24: put duplicati-b425eb436ce6747a4a947510cb260177b.dblock.zip.aes
6 Dec 2017 22:24: put duplicati-i1789202e3227431f9aaac2d1ddc92f19.dindex.zip.aes
6 Dec 2017 22:23: put duplicati-b15bd3f7ebc8b4f47ba8c21d24356d06b.dblock.zip.aes
6 Dec 2017 22:23: put duplicati-i8746ce40c2e1480cb71b5300736ea414.dindex.zip.aes
6 Dec 2017 22:22: put duplicati-b9b6498988c36454faac170b7b98a0c68.dblock.zip.aes
6 Dec 2017 22:22: put duplicati-id2676ec56946427aa578a65772bd32ac.dindex.zip.aes
6 Dec 2017 22:21: put duplicati-bfd1ce9111425479983ea43e64b345c70.dblock.zip.aes
6 Dec 2017 22:21: put duplicati-idf9e4eb1d8934e29961c10322e7ebab8.dindex.zip.aes

pcloud support assured me that they are not doing any rate limiting whatsoever. It also doesn’t appear to be related to encryption or compression because CPU usage is low throughout and if duplicati were busy with creating the file*, I would expect upload to cease entirely, but there is constant upload activity. I have also verified that the during the low speed upload periods, it is actually duplicati uploading and not something else:

image
image.png1167×351 23.6 KB

Swo what’s going on here? Why is the speed fluctuating so much? Should I complain to my ISP, or can it be duplicati that is causing this?

2

If you have the resources I’d suggest a test backup to a local network destination to see if the the same pattern emerges.

If so, and you’re using wireless, test again with a wired connection.

Lastly, test a local backup and see if you get the same pattern.

Hmm…maybe start with the local…

Let us know what you find and we can look further at fluctuation causes…

3

Looks like I can’t reproduce this locally. I’m getting a relatively steady transfer when backing up the same source (a NAS share) to my local harddrive (on the same machine where duplicati is running):

image

So this means my ISP is to blame?

4

As much as I’d like to say “ABSOLUTELY!” that’s not necessarily the case.

Aside from the “one result set isn’t pattern” issue, there are plenty of things that could affect throughput such as:

  • QOS (quality of service) settings on your router (VOIP phone or audio/video streaming call could be getting priority)
  • other uses on the LAN (Netflix, YouTube, print job, Windows update, etc.)
  • firewall (a software based SPI - stateful packet inspection - firewall could be trying to share CPU or other resources with the compression step so they both end up stuttering, though in your case I see CPU is low)
  • other devices on the LAN (do you have a smartphone that gets on your LAN? If so, it might be doing updates, email polling, etc.)
  • network technology (if you have cable you’re likely sharing bandwidth with the rest of your building/block, if you have DSL you’re likely sharing it with the rest of the people at your local substation, if you have satellite there’s a whole lot of atmospheric conditions between you and “them”)
  • provider (they may be throttling you)

Finding slowdowns in networks can be tricky, but it’s doable - let me know if you’re interested in tracking it down and I’ll try to remember some of my old tricks.

You could also try setting the --throttle-upload parameter to something near the bottom or mid point of your troughs and see if things even out. Not that you WANT it to run that slowly, but it might with later identification…

5

Apart from the probable suggestions that @JonMikelV suggest, it could also be related to a “small file” problem. If it is uploading the .dindex file, and there is some delay on the server when storing the file, you will see a performance drop.

1 Like