Backblaze upload extremely slow

Support
1

I know there have been similar topics, but I could not find a solution in there :frowning:

I am trying to backup to Backblaze B2 for the first time. There ist quite a lot of data (> 200GB) and the upload goes extremely slow (around 1,5 MBit/s).

My general upload speed according to Speedtest should be around 50MBit/s. The Backblaze speedtest shows about the same.

What can explain this sluggish upload speed? I have already waited days for the upload to finish after with it got stuck on “Waiting for upload to finish” for about a day, after with I restarted the docker container.

Block size is kept at default (50MB). I don’t think I have changed any of the default presets.

I run Duplicati in a docker container on my Synology NAS.

2

You won’t see maximum upload speed with Duplicati, as there is a lot of processing going on behind during a backup job: chunking of data, deduplicating, compressing into volumes… during this time a bunch of sqlite database activity, etc. The CPU in the NAS will definitely be a factor. What model Synology do you have? Have you applied any CPU or RAM limitations on the docker container?

By the way, I would avoid restarting Duplicati (or the docker container) unless you are certain it truly is stuck. You can watch the Live Log to confirm activity. About → Show Log → Live → and set the dropdown to Verbose.

3

Thank you for you answer.

I have a DS918+ which is one of the more powerful models. The CPU goes up to 70-80% while a backup is running. Is that really something that will limit upload speed?

Sure I won’t get my fill bandwidth worth. But even getting to 10MBit/s would speed the process up enourmously at this point.

I have no limitations on the docker container. I didn’t want to restart either, but the log had been stuck as well for more than 24h. I didn’t know what else to do.

4

Duplicati does not report MBit/sec anywhere. It reports MByte/sec. Are you converting value to MBit/sec?

That’s the verbose live log? Profiling is a higher level, but does Synology have tools to see process stats?
For performance problems, including seeming inactivity, it’s useful to see things like level of CPU and I/O.

I don’t use Docker and I’m not sure what tools are there (too few, I think), but you can run host tools, right?
For example, is the 70-80% CPU usage happening now with the stuck log, and can you see what uses it?

Basically, I’m not clear on the current situation. Did it move from slow to stopped? You can also look at the About --> System information screen in Server state properties to see if lastPgEvent moves.
You should be able to see its speed during uploads, although I suspect it’s the same as status bar speed.

If it’s only slow, you can try increasing parallel uploads, although by one report there was an error past 10.

  --asynchronous-concurrent-upload-limit (Integer): The number of concurrent
    uploads allowed
    When performing asynchronous uploads, the maximum number of concurrent
    uploads allowed. Set to zero to disable the limit.
    * default value: 4

You can also look in /tmp (probably) to see if there are a lot of default-50-MB files in the queue for uploads.

EU data center far slower than US data center is a Backblaze employee saying that’s only bandwidth, and explaining why multiple upload threads helps their client. On the other hand, it claims 10 Mbit/sec/thread is their (presumably system-related?) limit, and you’re far from that. Still, it’s possible that more threads help.

EDIT:

If you look in /tmp, also look over the dates on the dup-* files, e.g. ls -lcrt --full-time /tmp/dup-*.
On the Database screen, you’ll see the Local database path (hopefully mapped to persistent host file).
You’re looking for signs (via updated timestamps – there are other ways too) that somethings still moving

5

No, I’m sorry. It says MB/sec, I meant MByte of course.

No, it finished the backup but got apparently stuck on “Waiting to finish”

I split the backup into multiple parts and started it again yesterday. At the moment it is running at about 1,7MB/s. I will wait for it to finish and hope it doesn’t get stuck again. Might take another day or two.

Yes, I can. Although I am unsure what tools could give me useful information here.

I will try increasing parallel uploads with the next part of the backup and see if that makes a difference.

Thanks for the suggestions @ts678. I will report back when the current upload finishes.

6

Maybe it was “Waiting for upload to finish”? If so, that’s not tne end of the backup. The phases include:

I don’t know why it would be able to upload, then all of a sudden stop. You could get an idea by watching About → Show log → Live → Retry to see if an upload (or uploads) starts and gets stuck. This is easy to see if doing one upload at a time. If doing default 4 (or more for speed), it gets hard to pair messages up.

Maybe things will just work next time… BTW, your uploaded amount should decrease after initial upload, because only changes are uploaded, so possibly an uncomfortably long first upload will not occur again.

top might be good for CPU. Apparently there’s even a docker top command. Duplicati runs under mono.
This is getting less interesting, as I’m getting the feeling that things were quiet during the “stuck” period, however that could be confirmed through measurements and file time stamps, if the need arises again.

7

Yes, it was! I was just too lazy to type it out. Sorry

So, here we are again, it’s been stuck at “Waiting for the upload to finish” for about 5 hours. The CPU has been down to 20% since then.

How long should I wait for this to finish?

Screenshot_20210702-062652_Firefox Beta
Screenshot_20210702-062652_Firefox Beta1080×1553 138 KB

8

Are you seeing any network traffic?

9

Does Synology ship with good tools? If so, I’d be curious in the answer, although with default 50 MB remote volume size, I’d think that uploads should be finishing up pretty fast, unless an upload got stuck. It’s too bad screenshot cut off an interesting part (Backend event), although one would really want to check all uploads.

Another possible method to see what’s moving or not:

Would it be possible to try a small test backup to see if it finishes? What’s odd here (to me) is that it seemingly was uploading, then it wasn’t, and right here is also kind of a special spot. Looking at the Backblaze web UI “should” show a series of new 50 MB dblock files, each with a smaller dindex file.

There will probably not be a dlist file near the end, as that looks like the point where upload stopped.
Looking at a few of my logs, when it announces the second “CommitUpdateRemoteVolume took”, a
couple more uploads still need to occur. If the Profiling log hasn’t moved, that doesn’t seem normal.

10

I haven’t found a good tool to check this yet :frowning: most command line tools don’t run on the Synology

I did that last week! A small backup (even tried with about 1GB) finishes without a problem

Here are the server state properties. I don’t think lastpgevent changed lately.

Screenshot_20210704-080820_Firefox Beta
Screenshot_20210704-080820_Firefox Beta971×1490 144 KB

And here is a longer version of the log:

20210704_080611
20210704_080611972×5804 646 KB

The backup is still stuck :frowning:

11

I have been able to check the network traffic using vnstat. There seems to be some traffic ongoing:

image
image587×646 16.3 KB

However, when I just looked at my Backblaze bucket, I could not find any files uploaded after the point duplicati stopped logging progress.