I’m not enough of a DB expert to comment, however one thing I worry about is my generally limited understanding of the underlying design philosophy of keeping relationships consistent despite what catastrophes (e.g. hard kills) may occur. Consistency (or at least repairability) need is also external, especially for ongoing backups. Maybe it’s less so for things like DB recreate that can’t be resumed.
Processing IIRC does dlist files first, so dangling block references will exist until dindex readings.
Perhaps this order suits Direct restore from backup files best (where one picks the version initially)?
Sorry I’m not familiar with the code internals. Most of my experience has been external observation.
Write-Ahead Logging and Enable write-ahead logging and memory mapped IO #4612 may fit in too, especially when talking about performance, and potentially temporary file sizes during DB recreate.
In terms of external files that don’t have a transaction model, there’s a state-tracking mechanism to hopefully make it possible to clean up whatever sometimes-partially-done work is at the destination.
Some of this cleanup, though, relies on the database which in theory has some aids to transactions.
Question is, did we use transactions correctly, especially for the concurrent processing that occurs?
Sometimes one thread must flush something it wants out, but does that break the plans elsewhere?
One challenge that Duplicati faces on memory is the extremely diverse size of systems that it runs on.
Maybe a challenging case would be a large backup running on a NAS, perhaps with Duplicati Docker (because Synology as of DSM 7 no longer will install the current version – Docker is the workaround).
An excellent person to bring into this would be the hard-to-find original author who also tried a rewrite. Unfortunately, it never emerged. I wonder if you would be a suitable person to try to pick up the effort?
It seems a huge waste to discard something possibly so near at least some initial level of “completed”.
Original author still seems short on time though, but as someone who’s done lots, you deserve an ask. Your willingness to dig in is fantastic. Maybe you’ll be invited to join staff (which is quite thin right now).
@kenkendk what do you think about this seemingly very relevant topic, and also recreate questions?