I am on Docker version Duplicati - 2.0.6.3_beta_2021-06-17.
Backup stopped working last Saturday. At first I was getting about 53 files with size inconsistencies. I went ahead and deleted those files from remote backup.
I then attempted to repair database and received “The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.”
I then attempted to delete and recreate DB from WebGUI. Everything was going well but DB recreation failed eventually. I don’t remember the exact error message, I can re-run it again but it takes some time.
When I try repair again I get “The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.”
If I do verify files, I get “Found inconsistency in the following files while validating database:” with a list of files. The list of files is huge… 57903.
I can do list-broken files, but then it gives me a list that is about 57903 lines long and obviously I cant manually specify to delete this many files.
So at this point, what are my options? Start from scratch? This backup is about 7Tb in size and it runs over Internet (VPN), which would mean it would take a very long time to do a full backup…
Do you have examples? Such problems can vary with Storage Type of Destination. What is this?
Do you have a list of the names, or at least observations on them? Were they dblock, dindex, or dlist?
Did you happen to save the old database? It could be a useful reference for reconstructing the history.
Did you notice how far GUI progress bar got, or by any chance see About → Show log → Live action?
The log can show file downloads at Information or Retry levels. Verbose level adds details of progress.
The final 30% on the status bar can be slower, and the final 10% especially so (downloads all dblocks).
I suppose you could also take a guess on whether your Internet connection could do 7 TB in time used.
Is it listing source files or the destination files (which have names containing dblock, dindex, and dlist)?
I’m not sure how meaningful this test is now. It uses database records, but original database is deleted.
Possibly going down some version of list-broken-files and purge-broken-files earlier would go better. Sacrifice whatever got lost but keep going. Having 53 files go wrong at once seems a lot though.
Recovering by purging files explains that plan, and makes a remark about “has a valid local database”.
It would be nice if Duplicati kept a copy of old DB during a recreate, but I doubt it does. You can look…
Even if it comes to that, it would be worth trying to understand how so many files developed wrong size.
Using which Duplicati Storage Type? If it’s Local folder or drive, what’s actually underneath that?
Here is what I got in my notes: 2System.IO.InvalidDataException: Found inconsistency in the following files while validating database: /Audio Books/Warhammer Novels Collection/Zane Grey/Young Lion Hunter (388)/Young Lion Hunter - Zane Grey.epub, actual size 277489, dbsize 0, blocksetid: 8
This is another one: 2022-06-10 17:20:25 -04 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-ia99f5453076a48a29a4a5117a5d45ab4.dindex.zip.aes is listed as Verified with size 18461 but should be 18445, please verify the sha256 hash “TPIcH0+YeMpnId3apiW+M/el+Fz+2npe7/kF4BIbrZQ=”
Unfortunately I did not.
It got almost all the way to the end. Then stops with error message.
Here is what I get when I run list-broken-files:
0 : 2/20/2022 4:56:59 PM (116862 match(es))
/Audio Books/Brian Herbert & Kevin J. Anderson/Battle of Corrin/._Cover.jpg (82 bytes)
/Audio Books/Brian Herbert & Kevin J. Anderson/Battle of Corrin/._Dune The Battle of Corrin 0001 of 99.mp3 (82 bytes)
/Audio Books/Brian Herbert & Kevin J. Anderson/Battle of Corrin/._Dune The Battle of Corrin 0002 of 99.mp3 (82 bytes)
/Audio Books/Brian Herbert & Kevin J. Anderson/Battle of Corrin/._Dune The Battle of Corrin 0003 of 99.mp3 (82 bytes)
/Audio Books/Brian Herbert & Kevin J. Anderson/Battle of Corrin/._Dune The Battle of Corrin 0004 of 99.mp3 (82 bytes)
… and 116857 more, (use --full-result to list all)
SFTP (SSH). It runs over Internet (s2s IPsec VPN) and uses SSH. At remote location, it is a raspebrrypi4 with an usb external storage drive (it has its own power brick) mounted with NTFS extension. Overall the only time I see some issues is when internet becomes unstable.
These are very different sorts of files. You can’t delete source files from the remote backup, unless you mean you used The PURGE command. The source file names don’t exist at the destination. If the size errors were in dindex, dblock, or dlist files, you might have deleted them but it’s best to ask beforehand.
Moving or renaming files is sometimes better than deleting because then the files can be tested further such as attempting to decrypt. If file won’t decrypt, it’s damaged, so maybe DB (now gone) had it right. There’s a small chance a file (now gone) could be examined by hand or maybe even truncated-to-size.
means that the file either uploaded more than expected or grew, assuming 18445 was actually correct. Neither one is readily explainable by me for SFTP or your Linux NTFS, but I know little about the latter.
The original database would have had file upload history and directory listing sizes to add further clues. Even more reliable than databases (and might be good for future use) is a log file at at least retry level.
It probably had a dindex file that it couldn’t read, e.g. the one you cited might have become unreadable. Retry level on the log will show it doing retries, but if it’s actually bad at the remote then they won’t help. Recovery mechanism for blocks mentioned in a dlist file (which lists the blocks of all the source files), if dindex files don’t cover what dblock the block is in, requires downloading dblocks in search of the block.
That’s a lot of broken source files. I wonder what fraction of your total it was? These are ordinarily files affected by, say, the loss of a dblock file that had one or more blocks that were a part of the named file. Normally it relies on the intact database, and I’m not sure what happens when the database is bad too.
Unstable can sometimes be ridden through by increasing number-of-retries and/or retry-delay, however simply restarting the backup (next regular run) is supposed to upload a synthetic version of however far interrupted backup got, then keep going, so it might be OK to just let it give up rather than try very hard.
There’s not “supposed” to be any file corruption like files getting longer. Shorter is plausible, but is also protected against by not considering file to have uploaded until it’s listed at least once at correct length.
I suppose I could ask if you recall any oddities happening just before the events of original post began?
Would you happen to have size history on the remote so as to know what fraction of files were deleted? Sometimes it’s possible to manually trick Duplicati into attaching to dblock files that are already present. Possibly there are drawbacks, but one advantage is it would avoid uploading some fraction of that 7 TB.
By the way, Duplicati does best with not more than a few million blocks in a backup. Did you raise your Options screen Remote volume size to 5 MB or so? If not, starting fresh would be a good opportunity.
Either we work very hard at trying to figure out how much can be salvaged from however many versions you have, or if the main need is current versions not old ones, either start fresh with suitable dblock size, or keep dblock size and try to reduce upload amount by salvaging blocks into backup of the current files.
If you’ve got a whole lot of extra space, and also like old versions, I suppose you could keep the current destination files around for use with Duplicati.CommandLine.RecoveryTool.exe which is more tolerant…
Yeah I am trying to think if it is worth messing around with this or just start a new backup.
Duplicati states that I have 14 versions of back up, but when I go to restore, it only give me one backup option…
If there is a simple way to fix this and keep 1 version of backup, it would be great. If not, I guess I will look into recovering backup device from remote site and running initial backup locally (not over the internet).
If you mean 14 on home screen, that’s only updated when backup completes. Might that be the issue?
Restore screen goes directly into a different (and now recreated) job database, so I would trust it more.
If you want to look at the destination and count remaining (undeleted) files with names duplicati-<date> which would also contain dlist in the name (so they’re known as dlist files), they are backup versions. That’s the maximum number of versions left, and if some of the dlist files are bad, there might be fewer.
“Simple” is fresh start. “Faster, but less simple” is trying to reattach current files to the uploaded blocks.
One thing that’s different from previous ideas such as summarized here at bottom is you already ran a Recreate, so you might already have populated the database with whatever blocks were still surviving. Possibly your download speed is faster than upload, so the big transfer you ran wasn’t quite as awful?
was from an actual run of the reattach-blocks plan. There were some miscues in the action above that.
For your case, we’re trying to invent a variation which might start with setting no-auto-compact to make sure that doesn’t run and discard our blocks (and hope it hasn’t already…), then solve the dlist mystery.
If you truly only have one destination dlist file, its name should match the Restore date except it’s UTC. Download the file for backup, and because it may be used later Then set allow-full-removal so you can delete the version on the Restore dropdown. You said there’s one. The number on the left should be 0.
Use the delete command in GUI Commandline to delete --version=0 (set Commandline arguments).
You should now have a database of a bunch of old blocks, with no source files recorded as using them.
I’m not sure if we’ll need the empty dummy dlist file, because your Recreate had an actual one, so just running a backup might look at its old blocks and reattach what it needs. If this plan is simple enough, I could probably test some. Unknowns include whether remaining wasted (unused) blocks at destination would get cleaned up eventually, and whether lost dindex files will ever come back. I suspect they don’t which would mean your next Recreate or Restoring files if your Duplicati installation is lost may be slow.
If this has appeal, I can test it more, but there are other things that might be “off” due to current damage.
Fresh start is definitely a more certain process, and if you like you might be able to do it in chosen folder order at first, so that it goes in small increments with the most important folders first. GUI can deselect a folder that is currently selected by clicking on it. Sometime later, you can undo it to get folder backed up.