Database Repair UI is dangerously misleading

UX
1

I’m not sure if this is better here or on the githib issues page… if there is better, please let me know.

When a user selects the “Database…” choice under “Advanced” for a backup configuration, the UI shows text reading:

If the backup and the remote storage is out of sync, Duplicati will require that you perform a repair operation to synchronize the database. If the repair is unsuccesful, you can delete the local database and re-generate.

(“unsuccessful” is misspelled) and buttons labeled:

Repair Delete Recreate (delete and repair)

are available.

Everything about this implies that these are options for maintaining the local database. Yet, unless I misunderstand, when a local database is present, “Repair” updates the remote storage to match the local database. If no local database is present, “Repair” creates a local database from the remote information.

So the same “Repair” button does two very different things — modify remote storage to match the local database, or create a local database using the information in remote storage — neither of which is repairing the local database. (If there is any way to do that — to bring the local database into sync with what’s available in remote storage without deleting and rebuilding it — I don’t see where it’s done.)

My suggestion is that “Delete” is fine as it is; “Recreate” should mean “delete the local database if there is one, then rebuild it using the information in remote storage” and be enabled regardless of whether a local database is present; and “Repair” should be enabled only when a database already exists; it should be moved to a separate line and labeled something like “Update remote file lists to match local database” so it is clear that this option modifies the remote backup, not the local database.

I’m unsure of the best wording for that last one, because I don’t know exactly what it can and can’t accomplish. Obviously, if files are missing from the remote store which contain blocks from older generations of files that have since been changed or deleted, it can’t fix those. I understand that it does something that might stop Duplicati from throwing error messages during a backup; but what it really accomplishes and what is left undone (and possibly unflagged until some other operation encounters the error) is unclear to me.

2

The original author of repair is unavailable, but I think it goes both ways and can also repair the database.
https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/Database/LocalRepairDatabase.cs shows some examples of this, but I won’t claim to be familiar with everything, and my SQL skill is limited.

This kind of thing can be put in as an issue, and maybe even get fixed because it’s fairly trivial. The forum doesn’t track issues, so the chances of a developer finding them in the mass of posts is not all that good.

I would claim that recreation of a missing local database qualifies as an extreme form of database repair.
Later in this post I will raise the question of how limited or comprehensive the work of a repair should be.

This might get beyond UI and into changing The REPAIR command which is part of the split behavior:

Tries to repair the backup. If no local database is found or the database is empty, the database is re-created with data from the storage. If the database is in place but the remote storage is corrupt, the remote storage gets repaired with local data (if available).

The UI is actually a little different because (as the button notes), a Recreate will specifically delete the database. IIRC a manual Delete will disable the Recreate button. How’s your JavaScript? Mine’s poor, however it looks like this deletion is checked, and so a delete of a non-existent file should be avoided.

Still, there must be code somewhere to dim that button, so perhaps Recreate can be redefined to be constantly active, and to only request the delete if the database exists. In either case, do its recreation using the repair functionality and force it down the recreation path by ensuring database is not there…

Similarly, a UI developer (if one exists – and it sure isn’t me) might be able to move the dimming code
onto the Repair button, but saying it only modifies the remote is probably just not what it actually does.

This would make the buttons diverge further from the repair command, but it might make more sense.

I’ve found this area confusing due to its multiple behaviors, and asked about its capabilities previously, however I understand that internal code logic (e.g. SQL work) doesn’t easily translate to user meaning. There’s also sort of an argument that “repair” should be a broad concept without forcing tool selection.

The LIST-BROKEN-FILES command and The PURGE-BROKEN-FILES command must sometimes currently be run to repair a problem. These alter both the DB and remote, but are not in Repair button.

Although I think the UI is confusing and the underlying code rather vague in which side it will change, I suspect we’re stuck with the underlying code for awhile. A rewrite was attempted but it didn’t emerge.

The most dangerous scenario I know of is when people retrieve an older database from somewhere (perhaps an image backup) then repair, resulting in the inconsistency being “fixed” by trying to delete unknown newer information from the destination. My fix proposal is here, but isn’t getting discussions.

Ultimately Duplicati very much encourages volunteers in development and all areas, but there are few.
This severely limits how much progress can be made in doing new features or even UX improvement.

3

That is more or less what I did that got me started looking for answers.

I’ve been using Duplicati since 2017 to back up two critical folders (in separate configurations) to some spare space I have on my web hosting account. They’re small folders (Source: 7.23 GB / Backup: 8.04 GB / 412 Versions and Source: 9.96 GB / Backup: 10.69 GB / 206 Versions), so I keep unlimited history; easier than trying to figure out what I need to save “just in case.”

Earlier this year I opened an account with rsync.net so I could store some larger backups. I decided a few days ago that it would make sense to duplicate the folders on my web hosting account on rsync.net as well. As rsync.net makes it possible to run rclone, I was able to copy the backup folders easily. I run Duplicati backups nightly, and these two normally take less than five minutes each to complete, so I just planned to create two new backup jobs pointing to the same local folders but different remote folders.

I looked for a “new copy of backup configuration” option; since I didn’t see it, and I knew what was in those configurations, I just created new backup configurations pointing to the new remote folders and the same local folders. I considered finding and copying the local databases, but I wasn’t certain that was OK — I figured they might contain information specific to a given configuration, and something might get crossed up.

Well, I guess creating a new configuration created a new, empty database… or else I did something else that I don’t remember. In any case, I ran “Repair” thinking (as the text on the page implies) that would synchronize the local database with the remote folder. Instead, it deleted all but, I think, four dlist files in the remote folder.

Fortunately, since this was a copy of the remote folder on my web host, no harm was done — but it doesn’t make me happy to think how easily I could have done that to a backup set without a duplicate. I copied again, and this time for each configuration I did a Delete of the local database and then immediately attempted Repair. That appears to have worked. (There is something wrong with one of the two backup sets; I can successfully restore the most recent version of the data from it, but there were error messages when recreating the database. I haven’t yet been able to decode what they mean — I assume some of my history is lost, but I have no idea yet how to figure out what is missing or damaged. I’ve verified with rclone check that the rsync.net remote folder is an exact copy of the remote folder on my web host, so I know that wasn’t the problem. Whatever is wrong is wrong in my original copy as well.)

I very much concur with the original poster in that thread: however it needs to work, it shouldn’t work the way it does now. I see the sense of your idea of a heuristic that blocks likely bad use, but I really think the problem is simpler than that.

The more I think about it, I’m trying to figure out under what conditions a “repair” (that is not a “recreate”) will ever do what a user would want it to do. Also — especially given that repair is being run because there’s a problem, and it’s never far from likely that things won’t go quite as expected — why on Earth is repair deleting potentially unrecoverable data, instead of moving those files to a subfolder of the remote folder, from which they can then be deleted if all goes well, or recovered if it doesn’t?

4

Since the remote collection is supposed to be restorable on its own as I understand then this could maybe be warranted.

Its more debatable if it were otherwise as at anytime updating the remote db could in theory become broken anyway and you can’t really have two versions here anyway (like game saves) as the backups themselves would need to be fully copied.

Either way, validation should - if its working properly - rather invalidate the need to do anything here anyway.

Since any changes to the backup whether its the db or whatever could break restoring. So anything that fails in either may be overcome via code and allow restoring of whatever else in there that works.

Validation and the code fixing anything wrong is the means to keep it it working. That doesn’t validate or invalidate the need to work on a temp file, the button use here, or whatever. Temp files are used to keep certain issues away but they can’t fix code problems and code can fix problems.

They could though indeed write all changes to temp on the target and then rename or append to the files or whatever if they don’t already. But, it doesn’t stop code problems. If the new data is messed up, the target backup is still borked.

I don’t think there’s anything here that’s major though. It would be better to work on validation and fixing the db and backups when issues show up.

The normal user (99% of all users) only needs it working, not how it works. If it always works, then no one needs to know anything. In reference to button meaning itself.