Failed to repair a database

ubenmackin · March 15, 2018, 7:40pm

Recently one of backups started failing. I decided to try and recreate my local database to make sure things were in order. When I run a recreate it runs for 12+ hours but then it eventually fails:

Failed to process index file: duplicati-iea25543756e14548add40ac3c3ffabea.dindex.zip.aes
Duplicati.Library.Interface.FileMissingException: The requested file does not exist
  at Duplicati.Library.Main.AsyncDownloader+AsyncDownloaderEnumerator+AsyncDownloadedFile.get_TempFile () [0x00008] in <533fc6a2ce6a49e5868bdd6f335ced4e>:0 
  at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00b24] in <533fc6a2ce6a49e5868bdd6f335ced4e>:0

So I tried doing a repair:

The database is marked as "in-progress" and may be incomplete.
Found 6 missing blocklisthash entries, repairing

System.Exception: Missing block for blocklisthash: aCKfgS7vTUo8zpylWMHAiogs5wyL52sGtZ9e+uc3xYg=
  at Duplicati.Library.Main.Database.LocalRepairDatabase.FixMissingBlocklistHashes (System.String blockhashalgorithm, System.Int64 blocksize) [0x00246] in <533fc6a2ce6a49e5868bdd6f335ced4e>:0 
  at Duplicati.Library.Main.Operation.RepairHandler.RunRepairCommon () [0x000ce] in <533fc6a2ce6a49e5868bdd6f335ced4e>:0 
  at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00147] in <533fc6a2ce6a49e5868bdd6f335ced4e>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass21_0.<Repair>b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in <533fc6a2ce6a49e5868bdd6f335ced4e>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00072] in <533fc6a2ce6a49e5868bdd6f335ced4e>:0 
Return code: 100

So I’m stuck now. I can’t do a repair, I can’t do a recreate, and I can’t run a backup. Any thoughts on how to get unstuck? The backup is 1.2 TB, so I’m hoping to not have to start over.

JonMikelV · March 20, 2018, 2:07pm

Hi @ubenmackin, welcome to the forum!

Sorry for the delayed response (stupid real life) - did you move forward with this yet?

If not, I have a few quick questions:

what version of Duplicati are you running?
what OS are you running on (Windows, Linux, MacOS…)?
is it possible you ran out of disk space in your temp folder while the rebuild was happening?
is the 1.2 TB size you mentioned how big the SOURCE is or the backup?
do you know how big your .sqlite database is?

As far as not having to start over, the only real option is to try the rebuild again. We know rebuilds can be slow (it’s up there on the to-do list) but it’s the only way to be able to do backups on a “broken” job.

You should still be able to do restores, so your backups are lost - you just can’t add to them them until the rebuild is done.

When a rebuild happens a lot of the destination content has to be downloaded so it can be “scanned” back into the local database form. The error you go seems to have happened in the downloading block for just as it said - a file was downloaded then couldn’t be found.

Oh! If you could check your logs and see if that index file that failed to process was attempted to be downloaded more than once, that might help narrow down where things went wrong…

ubenmackin · March 21, 2018, 1:59am

No problem. I am still having issues.

Duplicati - 2.0.2.21_canary_2018-03-06

Linux

Not likely, there is approximately 150 GB free space on that drive. I’ve tried a few subsequent rebuilds, and they all keep failing, and there appears to be plenty of space.

Source is 1.21 TB. The Backup is the same size as the source, as I am only keeping 1 copy.

I actually just deleted it to do a rebuild again, so right now it is 850 kb. I think it was about 4 GB before I did the rebuild.

Ya, I’ve tried this a few times, and it keeps failing because of the “The requested file does not exist”.

How best can I check the log? Do I do it form the web gui or from the command line?

JonMikelV · March 21, 2018, 8:53pm

If the error is repeatable I’d suggest adding ‘–log-file=[path]’ and ‘–log-level=profiling’ to the job and run it.

This will log stuff to a file at [path] so you can view it easily. Note that this adds to the log file with each run so be sure to turn this if when done or you’ll eventually have a giant log file (trust me).:

Otherwise you can use the GUI. The job menu had a Show log entry and the global About menu had one as well. In the global one of you choose the Live log in suggest using the Profiling setting to get the most detail with your version.

If this doesn’t help us figure out the issue then I think we can manually adjust the database to remove the bad blocklisthash references.

ubenmackin · March 23, 2018, 9:19pm

Sorry it took awhile to get back to you. It looks like it tried 10 times to get a file:

Mar 23, 2018 2:16 PM: Backend event: Get - Failed: duplicati-bcec1539e78c64ec783034eaf1181a6b2.dblock.zip.aes (1,023.96 MB)
Mar 23, 2018 2:16 PM: Operation Get with file duplicati-bcec1539e78c64ec783034eaf1181a6b2.dblock.zip.aes attempt 10 of 10 failed with message: The requested file does not exist
Mar 23, 2018 2:16 PM: RemoteOperationGet took 0:00:00:28.470
Mar 23, 2018 2:16 PM: ...
Mar 23, 2018 2:15 PM: ...
Mar 23, 2018 2:15 PM: Backend event: Get - Started: duplicati-bcec1539e78c64ec783034eaf1181a6b2.dblock.zip.aes (1,023.96 MB)
Mar 23, 2018 2:15 PM: Starting - RemoteOperationGet

And when I expanded the one “failure” entry, here was the details:

{“ClassName”:“Duplicati.Library.Interface.FileMissingException”,“Message”:“The requested file does not exist”,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:" at Duplicati.Library.Backend.OpenStack.OpenStackStorage.Get (System.String remotename, System.IO.Stream stream) [0x00090] in <10f599241a0348c998ef9c336e548983>:0 \n at Duplicati.Library.Backend.HubiC.HubiCBackend.Get (System.String remotename, System.IO.Stream stream) [0x00000] in <31c595d068ba469790b05db6c51e01bb>:0 \n at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x001a2] in :0 ",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:null,“HResult”:-2146233088,“Source”:“Duplicati.Library.Backend.OpenStack”}

and then after a few more minutes it fails. Here is the stack trace from the failure:

{“ClassName”:“Duplicati.Library.Interface.FileMissingException”,“Message”:“The requested file does not exist”,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:" at Duplicati.Library.Main.AsyncDownloader+AsyncDownloaderEnumerator+AsyncDownloadedFile.get_TempFile () [0x00008] in :0 \n at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x0101d] in :0 \n at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run (System.String path, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00037] in :0 \n at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal (Duplicati.Library.Utility.IFilter filter) [0x000ba] in :0 \n at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00012] in :0 \n at Duplicati.Library.Main.Controller+<>c__DisplayClass21_0.b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in :0 \n at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x000ed] in :0 ",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:null,“HResult”:-2146233088,“Source”:“Duplicati.Library.Backend.OpenStack”}

ubenmackin · March 28, 2018, 5:09am

@JonMikelV would you mind sharing how to do this?

JonMikelV · March 28, 2018, 6:58pm

Sorry, I was using the forum “we” - I don’t personally know how to do this, but I have seen posts by others that have done so.

But that’s a pretty drastic step and now that I’m seeing the error I think there might be something else going on. In your post of the error the first line is:

Mar 23, 2018 2:16 PM: Backend event: Get - Failed: duplicati-bcec1539e78c64ec783034eaf1181a6b2.dblock.zip.aes (1,023.96 MB)

This gives us two things to look at - the first is, can you look at the destination and verify that the file duplicati-bcec1539e78c64ec783034eaf1181a6b2.dblock.zip.aes really does not exist?

The second thing I’m seeing is that the file appears to be about 1G is size - are you using a dblock-size (Volume size) of 1GB?

ubenmackin · March 29, 2018, 1:55am

Ah, no problem

Just checked, and it looks like the file is there. I even tried downloading it, and it downloaded just fine.

I am. Since my total backup size is 1.2 TB, I figured 1 GB dblocks would be appropriate, because then there would only be about ~1,200 blocks on the server. If I did the default of 50 MB blocks, I would have had ~24,000 files. I thought that seemed like a large number of files.

Was that a mistake to use blocks that are so large?

ubenmackin · March 29, 2018, 1:58am

Just to add. If I try a repair at this point, I get the following error:

The database is marked as “in-progress” and may be incomplete.

System.Exception: Missing block for blocklisthash: aCKfgS7vTUo8zpylWMHAiogs5wyL52sGtZ9e+uc3xYg=
at Duplicati.Library.Main.Database.LocalRepairDatabase.FixMissingBlocklistHashes (System.String blockhashalgorithm, System.Int64 blocksize) [0x0024e] in :0
at Duplicati.Library.Main.Operation.RepairHandler.RunRepairCommon () [0x000dd] in :0
at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00167] in :0
at Duplicati.Library.Main.Controller+<>c__DisplayClass21_0.b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x000ed] in :0
Return code: 100