Repairing not possible, how to proceed?

Support
1

Duplicati - 2.0.6.105_canary_2023-04-09 on Debian, ssh backend

My long going backup failed due to some inconsistency issues and I tried to delete and recreate the Database: with the following result:

  1. Apr. 2023 21:03: Die Operation Repair ist mit folgenden Fehler fehlgeschlagen: Found inconsistency in the following files while validating database: /home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/alice.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142 /home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/bob.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142 /home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/imgbackup/empty.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142 /home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/empty.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142 /home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/empty.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142 … and 8 more. Run repair to fix it.

{“ClassName”:“System.IO.InvalidDataException”,“Message”:“Found inconsistency in the following files while validating database: \n/home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/alice.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142\n/home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/bob.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142\n/home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/imgbackup/empty.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142\n/home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/empty.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142\n/home/skrodzki/work/qsol/git/syncosync/amd_installer_emulation/bullseye/empty.img, actual size 524288000, dbsize 393216000, blocksetid: 1121142\n… and 8 more. Run repair to fix it.”,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:" at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x000e9] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x01496] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run (System.String path, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00037] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal (Duplicati.Library.Utility.IFilter filter) [0x000ba] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00012] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Controller+<>c__DisplayClass18_0.b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x0026f] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x00007] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Library.Main.Controller.Repair (Duplicati.Library.Utility.IFilter filter) [0x0001a] in <985e6d8a4fea4b938496104eb1abdc02>:0 \n at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x003ad] in <1fb2f5acbba047c8b281d2f7d847aa06>:0 ",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:null,“HResult”:-2146233085,“Source”:“Duplicati.Library.Main”}

Is there any chance to repair / recover anything?

2

This seems to be the value that your system is setting for several files that have all for length the max default value (524288000), so it can be assumed that the latter is the correct one (that is, the one that is read from disk), however what’s written in the database is differing by only a few bits.
So in my opinion, there is a slight possibility that your computer has a defect that is flipping bits at some points either in RAM or in the disk controller.

Needless to say, it would be a serious problem and a good idea would be to stop using the system completely and use another computer to test the backup by trying to recreate the database (if it was my computer I’d setup a read only user account on the sftp server for that because in this case you may have corrupted files on your system and old backups versions would be very valuable).

3

Hmm… I am a little bit confused: does the repair database at any point read the actual data from my machine which is backuped? I had the understanding that it only reads data from backup destination to rebuild the database?

Anyway: regarding those files which show up, these are sparse files. Could this be the reason for the size difference?

On my actual filesystem (so the real size could differ, the sparse size is the same):

stat --format=%n:%s alice.img 
alice.img:524288000

du alice.img 
308072	alice.img

but even with that, the backups where working for the last 3 years :slight_smile:

4

Can you describe that issue any? Any of the same files named?
Topic title says “Repairing not possible”? Any other description?
Was the Repair attempted before or after the Recreate, or both?

Unfortunately the code has a hardcoded limit of 5, but the ones shown look very repetitive down to the blocksetid, which means the content is the same. Is this expected, and might the other 8 be same too?

If you want to go look in this database, I can guide you a little. Or you could post a link to a bug report.

It should be possible to avoid upload of data by reattaching blocks to current source, but it loses history which you might not like. By recover, are you trying to do a restore, or do you mean save some history?

Fixing this some other way may or may not be possible, but you could certainly first try easy ways in the hope that something works. For example, the message suggests Repair. Using Commandline to run the list-broken-files command might not find anything but is easy to try. Some have used purge, but it needs file names and we don’t have them all. It’s also sometimes blocked by other issues. You could test a file.

5

Another history question would be whether the .img files change?
Anything known that might have led to whatever old problem was?
Sometimes it’s possible to fix issues by deleting affected versions.

6

Unfortunately I have not noted the issue when doing the backup, but I am pretty sure, it stated an issue with the database - which happens from time to time - and therefore I chose to recover the database be UI (delete and repair)

With repair not possible, I mean the repair of the database… I have not used the CLI until now, but could of course do so…

I guess, I “know” all the files from this issue, still I am wondering, why it appears now, they are backuped since years this way.

Saving some history and of course understanding to work with Duplicati. A few months ago I tried for test reasons a complete restore without database for my 300Gig Home, which took around 2 days… but it was a success at the end…

Trying to list-broken-files does not work, as the cli says: ErrorID: CannotListOnDatabaseInRepair
Cannot continue because the database is marked as being under repair, but does not have broken files.

I have not tried to purge the files as I guess this will also lead to the same result…

7

yes, they change. I think I could save a lot of space by excluding them from my backups, these are just qemu images for my CI/CD chain…

but now … how could I do anything with no repaired database?

8

I’m pretty sure you could create a bug report, as the reason it exists is to look at issues in databases…

You might also be to delete, e.g. version=0 is the latest, but we don’t know how far back damage goes.

Do you recall how far on the GUI progress bar the Recreate got? Past 70% it gets slow and reads a lot.
For the best view, you’d watch About → Show log → Live at at Verbose level to watch progress reports.
If it went tolerably well, you could probe for undamaged versions by recreating DB for specific versions.
You can normally use the Restore version picker to see the dates and the version numbers (on the left).

EDIT:

Commandline delete is how to delete a version. A limited recreate uses database Delete button and the repair command with a version option. You might consider copying your current database first, as it has useful information in it if it can be examined. Same was true of database before past Recreate. Too late.

9

You can also look in the database yourself with DB Browser for SQLite, often in repo as sqlitebrowser, however it would probably just be looking, not editing. Looking in destination files would be an option too. There’s possibly a dlist file with some inconsistent information. Ideally, the older ones would be correct, meaning if the delete command won’t cooperate either (needs a test), one could just move the bad dlist files elsewhere (or rename them to not start with duplicati-) to recreate with good ones if old are good.

10

Ok, I will do so.

At the moment I can not dig deeper inside as I have sufficient backups working with other backup software. I wanted to understand if I could do anything “easy” to repair my database. From a users point of view, I’d expect the DB repair process to leave out broken files on the backup to still provide access to most of data possible. Also as far as I know, there was since years the discussion to backup also the database itself - which I did for some time but stopped doing so, feeling myself to secure.

Maybe the bug report helps to fix this issue for other people using Duplicati.

Thanks a lot for all your help!

11

The first step to improving the code is understanding the situation, ideally to find a reproducible case. Typically the next step is hard too – finding someone willing and able to code a fix for a rare problem. Duplicati has many more issues on record than there are developers, so prioritization has to be done.
Database repair fails, recommends repair to fix, so this goes in circles? #4923 is yours with this topic.

Seeing this bug report might suggest what sort of problem in the dlist or dindex files could cause that. Confirmation could be obtained by looking in the file, but one challenge is that a dlist file could be big, meaning hard to fit into an editor without exhausting memory. Another challenge is that it’s a long line.

Thanks for helping push it one step further.

12

I have sufficient backups working with other backup software.

Can I ask what other backup solutions are you using as a secure?

13

Thanks for the bug report provided here on your cited GitHub issue. It was immediately apparent what the check was complaining about, and totally unclear how things got that way. It might be possible to discover more in the Destination files, but some of those are sensitive (especially dlist and dblock. dindex are less).

I’m not quite sure how to proceed, but I did think the forum is a better discussion forum than GitHub issue, and topic has now been awakened with another question, so that’s another reason for me to discuss here.

14

On Linux I go with duplicity/deja-dup, on other systems I use restic. I’d love to use and recommend Duplicati as the UI and also OS-Integration (autostart, scheduling) is IMHO superior. But TBH I had too often troubles with corrupted databases or infinite db recovery time… And the most important task for a Backup Software is to backup and restore reliable…

1 Like
15

I thank you very much for your reply.

I really discovered Duplicati recently and it’s great, what I was looking for for a long time. Easy to use, you can protect the interface, schedule backups, etc…
So far I haven’t had any problems but I’m worried about reading about the problems with the databases since my idea is to start using it with some small clients. (No more than 500 GBs)

I will continue to investigate how to protect or backup the database.

Restic I am going to investigate how it is used in Windows. But there is very little documentation about it.

If I could I would make backups with Duplicati and Restic in Windows.

Thanks again