Why is duplicati SOO slow?

Support
1

I don’t want to sound like an ahole but why is duplicati slower than molasses? It’s taken almost 4 days to upload 5TB to backblaze. Running the same test with synology’s backup app took around 28 hours, and duplicity also took around 26-29 hours.

Details:
Internet: 2.5Gbps fiber
NAS running duplicati: Ugreen DXP6800 Pro (12 cores, 16GB of ram)
NAS Storage: 6x WD reds in raid6, 2x 256GB NVME drives for r/w cache

It has taken roughly 5 days or so now to offsite 5TB of data. When it runs, it seems to be decently fast, when it show a status, it shows between 50MB and 200MBps, but it seems to pause. It’ll churn through a few files, look like its doing something, pause for 30-60 seconds pause, etc.

image

I understand it has to create a zip, then upload it, but it should not be this slow.
My job settings:

5 days to upload 5.39TB is ludicrous. in best cases, it should take 6 hours. even accounting for the additional time to find, zip, upload, find, zip, upload, etc, even a generous 28-40 hours would be fine but 100+ hours? I doubt it’s the WD reds, as I’ve had them in a NAS before and they’re able to Read/Write at well over 100MBps, and I previously tried with a larger hash size of 10MB, as well as both 500MB remote volume sizes and 50MB remote volume sizes, as well as tuning the number of threads from 0 (dynamic) to 10, no difference.

Setting Value
Limit size of volumes: 200MB
Number of times to retry a failed transmission 10
Percentage of samples to test after a backup 4
number of concurrent uploads allowed 8
Block size used for hashing: 1MB
zip compression level 0
Number of concurrent threads 8
Skip Hash Checks Disabled
Determine if verification files are uploaded Enabled

Is there some other setting I’m missing?

2

Hi @erich81, welcome to the forum :waving_hand:

From that description it is not clear to me if what is causing the pause is uploads or processing being the limiting factor. Do you see evidence of one or the other?

It could be that Duplicati is doing something dumb with the transfer and that limits the sending speed. Can you see if the link is saturated (at least in bursts)? I assume you have not set up bandwidth limiting.

Do you see the same slowness for a smaller sample backup? Like 100GB?

3

I haven’t tried a smaller backup yet, but looking at the “bandwidth” tool on the NAS, its every 30-120 seconds, sometimes longer that you get a spike of 100-200MBps. then zero. The container is unbounded, meaning it can use as much cpu or ram it needs, but if i had to describe the behavior, it’s almost like it should be “scan, zip, upload, scan, zip, upload, etc. instead, its like “scan, zip, upload, sleep 30-300, scan, zip, upload, sleep….” etc

i see nothing in the logs to indicate what might cause the pause. once the initial sync is done (just completed my 3rd test after 3.5 days, sequential uploads are faster, but the initial sync is soooooo slow.

4

Yes, that is the general idea, but everything is running in parallel, so that makes it a little harder to observe what is really going on.

I think we can rule out bandwidth issues then, unless you are seeing failed transfers.
Failed transfers will cause a retry delay, and that could explain it.

Internally, Duplicati runs roughly this:

  • Enumerate files
  • Scan files, hash chunks
  • Compress new chunks
  • Encrypt filled volumes
  • Upload filled volumes

There is a write-ahead option so the scanning part can create volumes and that are queued for upload, and the upload queue will process these (also allows parallel transfers).

If we rule out the failed transfers, then it sounds more like the local processing is causing issues. On the second run, it is faster because it checks timestamps on known files, and if these are not modified, it skips scanning for new blocks.

So that means on in your machine, one of these are the problem:

  • Scan files: reads each file sequentially, hashing each block (1MB)
  • Compress chunks: writes new files to temporary zip (you have zero compression, so this is just copy)
  • Encrypt volumes

I will rule out encryption as it is (usually) quite fast and should also affect your second run, which it does not.

To dig into the scan part, what it does is:

  • Read a chunk
  • Compute the hash
  • Check if hash is known

The last step, check if hash is known, is done by making a query to the local SQLite database. If this database is on a rotating disk and the number of blocks is large, this could cause some latency in checks.

What is the size of the local database, and is it on a non-SSD disk?

5

the DB is on the RAID6 volume, so not SSD. I can move it but not at the moment, would need to rebuild the array, or break my RAID1 SSD cache to do it. I can try moving it if you think moving the DB will help significantly.

6

It depends on the size of the database. At some point the database will be too large to fit in memory, and since the lookups are for hash values, they are essentially randomly distributed and will hit all over the place.

If the pages are not all kept in memory, it will often cause cache misses. Any cache miss depends on the seek speed of the disk, so a sufficiently large database will benefit from being on an SSD.