FTP delete not atomic / verified / transactional (?)

[ts678]

@Sami_Lehtinen

Sounds strange and unlikely.

Search for “database disk image is malformed” at duplicati.com and Issues · duplicati/duplicati · GitHub
The problem with having 4 million backups per month run is that unlikely things arise every now and then.
How To Corrupt An SQLite Database File is SQLite’s official position, but diagnosing user issues is tough.

One time when I was buying a hard drive, I noticed an SMR drive’s data sheet not guaranteeing integrity if power was cut. I was avoiding SMR for other reasons, but I definitely don’t want any easy drive corruption.

@Sami_Lehtinen

logical high level corruption

This is more common. The “database disk image is malformed” is a low-level corruption of the database.

@Sami_Lehtinen

I would personally implement this in a way where:

Thanks for the thought. I don’t know how that compares to the current design. Feel free to look if you wish.

@Sami_Lehtinen

Sometimes, it’s necessary to create separate table, for “pending updates” just like there’s the queue directory and so on. Way too often developers modify the actual data directly, causing problems if commit is called (or something is rolled back).

This is one of my worries. Duplicati handles tables directly, and tries to control commit time, but a commit may come from a different thread. This is all my speculation though. I haven’t read through all of the code.

You can see the backup startup here, and Channel Pipeline helps understand the parts and the data flows.