Hello
I would like to share and ask duplicati developers and users the advanced options you use to get better backup performance.
At the moment I make use of these options, I thank the suggestions and opinions.
–usn-policy = Off
–snapshot-policy = Off
–thread-priority = highest
–check-filetime-only = true
–zip-compression-level = 1
Everything is a trade-off. Compression is perfect example, high or low compression can make things a lot faster or slower, depending on different factors. There’s no single right or wrong value available. it all depends.
Instead of compression and decompression slowdowns. I would focus on small I/O blocks. Quite often it seems that duplicati maxes out disk I/O, on IOPS, but with very litte actual data going in and or out. So basically it seems like random 4kbyte access. Which easily leads with spinning rust disks, to I/O rates around 1 - 4 megabytes / second. Getting higher throughput would be easy, if the disk access pattern would be different. I’ve been running single backup now for quite a while, and I haven’t seen it hit 10 Mbytes/s level even once. Even if the disk would do 80 - 100 Mbytes/s on sequential access easily.
Option to add I/O buffering would be awesome. When I do larger ETL jobs at work, I often buffer data to form read / write batches of 4 - 64 megabytes before invoking disk I/O. Just to keep the cheap large storage disks actually moving data and not getting stuck not seeking data.
Note, the data being backed up and restored in my case is huge database files, in range of hundreds of gigabytes to terabytes. Not any random small files. But even with those smaller files reading bunch of files in a batch and using memory to buffer writes would be beneficial.