Local Backup Intermittent Inactivity

Support
1

Hello Everyone, I apologize if this is a duplicate topic, I searched for a bit and found some tips to increase speeds but never seemed to get much better performance.

I’m backing up about 7TB of data from a local disk to another local disk. Standard windows file explorer copies/moves from the disks max out at about 150MB/s. There seems to be quite a delay in the creation of the volumes. There’s about 20 sec of inactivity on the destination disk before each volume transfer begins, then about 5 seconds for a volume transfer itself; destination disk speed goes up to about 150MB/s writes, then back to zero for another 20 seconds while the next volume is created. I’ve tried lowering the block size to near the defaults with the same speeds. I also tried to remove any limits of volumes created; instead it seems the software just can’t make them fast enough as there is never more than 1 volume waiting in the asynchronous-upload-folder. Overall the speeds calculate to about 30~40MB/s because of the delays. Is this to be expected with a fast local destination or is there any way to get the volumes to build faster so that there’s a constant build up of volumes to be sent to the destination? I suspect many don’t notice this delay as they use Duplicati to backup to external sources (cloud providers) over their internet connection; this would of course allow volumes to build up since most internet upload speeds are well under 150MB/s.

Here are my details:
running Duplicati - 2.0.2.13_canary_2017-11-22

Encrypted Backup using AES-256
Volume Size: 1GB
–asynchronous-upload-folder=F:\Duplicati-Scratch
–asynchronous-upload-limit=0
–blocksize=2MB
–disable-streaming-transfers=true
–zip-compression-level=0
–tempdir=F:\Duplicati-Scratch
–use-move-for-put=true

Appreciate any replies

2

It looks to me like you’re doing pretty much everything necessary to squeeze out the fastest performance. At 7TB I wouldn’t be surprised if you’re just running into general sqlite database performance issues now.

Unfortunately, there’s not much that can be done to speed that up beyond optimizing the SQL calls, possibly storing the database on an SSD or ramdrive, and/or restructuring the database itself. The first and last of those items are being looked at, but no major advances have been made yet.

3

Thank you for the reply, I figured as such. It’s not the end of the world as the data rarely changes so once the initial backup is done in a few days, any additional files (usually small batches 5~10GB a day) will be backed up very quickly.

4

That’s generally my experience - a “painfully slow” initial backup then fast runs.

5

2 posts were split to a new topic: Where are the sqlite databases stored?