Attention SSD users: if your temp directory isn't ram-backed, Duplicati will prematurely wear your SSD

[ts678]

@davide.gironi

Also, what’s about the use-block-cache settings? Does it helps?

No, that’s entirely unrelated although the name and documentation are both vague enough to be tempting.
Useing a RamDisk for the asynchronous-upload-folder is how I think it works (and what the name refers to).
use-block-cache does not work as expected; high SSD wear #3566 speaks of rename, docs, and a feature (however features have a long backlog – good part is that gives some time to start discussing beforehand).

@davide.gironi

high write usage also for the database file

One tradeoff which I think is especially visible on a database Recreate is that an SSD can help add speed. Unfortunately it also takes its toll on the SSD. I don’t think we would want the database kept in a ramdisk…

@davide.gironi

$ConvertToNonresident file

is a Windows internal file. Your image refers to “Paging I/O”. So does this article. Search can find other info.

@drwtsn32

Perhaps the ultimate fix would be for Duplicati just to store the files in RAM instead of on disk. If it stored directly in RAM there would be no need for a RAM disk.

Several (but not most) third-party backend access methods have file-based APIs, so can’t stream transfers. Possibly this would be kind of a sometimes-can-have-it-sometimes-can’t thing by the time the dust settles…

@John_Doe

It does seem a bit odd that duplicati uses a temp dir for the blocks that are getting ready to be sent over the network

This is probably partly for performance goals about keeping a steady upload, and the queue size is settable. There’s also the previously mentioned inability to stream files to some destinations, and worry about messes. Somewhere there’s a comment from the lead author to that effect, I think. At least initially, simplicity is good…

@NotHere

Duplicati does not clean always clean up after itself.

Of course it should but it doesn’t yet. Keeping messes somewhat confined to temporary files is a good thing.

@Pectojin

Actually, blocks are packed into volumes, which default to 50MB in size. So your numbers are a bit off, but it’s still a valid point that you could fit a couple volumes in memory if you wish

Some people have wanted huge volumes for large backups. Choosing sizes in Duplicati discusses tradeoffs. Having a roomy drive is more forgiving of configurations, and Duplicati temp dup-xxxx files not being deleted, however whether or not one thinks SSD endurance is an issue now, the shift from TLC to QLC will get worse.

@John_Doe

Watching the asynchronous-upload-folder, It looks like it holds the blocks, not the volumes in the folder. that’s why I was working with blocks and not the volumes.

It’s hard to tell them apart (certainly not using the random file names). On my system I generally get a mix of 50MB dblock volumes and smaller files that are dindex files (done after the dblock, I think) and a dlist at end. Look at Job --Show log → Remote for lines doing put to get file names. Click on these lines to get file sizes.