Backup corrupted

1

Hi Everyone,

I’ve been using Duplicati for several years to backup ~200K files totalling ~100GB to a Google Drive cloud backup, with a “7D:1D,4W:1W,12M:1M,100Y:1Y” retention policy (meaning I am doing frequent – daily – backups). I am using a dedicated Google account for the backup, so only Duplicati can modify the remote files.

As a good conscious guy, I regularly do a DRP test (once every 3 months).
My DRP test consists of (1) reinstalling duplicati, (2) rebuilding the database from the cloud backup storage, and (3) restoring the files.

Most of the times, it succeeds. But, unfortunately, every approximately one year it fails on step (2) as the database can not be rebuilt from the cloud storage (e.g. Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected).

I typically workaround this by deleting the remote storage and starting with a new backup from scratch, which is obviously not ideal (I expect to always be able to rebuild the database and restore from the remote storage system).

I’d like to work together with Duplicati’s dev team to solve this problem for good.

Right now I have a “corrupt” remote backup which we can use for troubleshooting. What steps can I take to work with the development team to identify the root cause and solve it?

I’m a computer science engineer with a software dev background, so I can manage detailed technical instructions.

Thanks,

Gaston

2

It looks like the problem was that initially my backup was created with 2.0.6.3 beta, but I tried to rebuild the database using the latest canary version (2.0.9.108).

After retrying using 2.0.8.1 beta, the database could be sucessfully rebuilt.

I’ll try to restore the data to confirm everything works, then I’ll do some more “divide and conquer” and report here.

For information, the error I got when trying to rebuild the database on 2.0.9.108 was:

Oct 20, 2024 12:39 AM: Failed while executing Repair "Gaston's profile (Cloud)" (id: 1)
code = Error (1), message = System.Data.SQLite.SQLiteException (0x800007BF): SQL logic error
no such column: Temporary
   at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, String strSql, SQLiteStatement previous, UInt32 timeoutMS, String& strRemain)
   at System.Data.SQLite.SQLiteCommand.BuildNextCommand()
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery(IDbCommand self, Boolean writeLog, String cmd, Object[] values)
   at Duplicati.Library.Main.Database.LocalRecreateDatabase.CleanupMissingVolumes()
   at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run(String path, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal(IFilter filter)
   at Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass16_0.<Repair>b__0(RepairResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Repair(IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)
3

Oct 8 fix will presumably be out in the next Canary after current v2.0.9.108_canary_2024-10-03. Canary releases by definition are new and untested, so beware of bugs. Testing help does help.

Tracking down your annual issue if it’s on 2.0.6.3 might just find it’s fixed since, but working with something more current may be useful. It may involve lots of attempts to preserve historical info.

Ideally what one wants is probably a GitHub Issue with exact steps for anyone to repro a failure. Actually a pull request would be better, but heading the other way, one collects data for the devs.

Suggestions from me are available, and I collect a lot, and try to avoid a latent-problem situation. Sometimes this means test scripting to do a more intensive check than you (or Duplicati) now do.

4

That exact error is created here:

The logic here is that it attempts to build a complete database that you can continue your backup from. A failure here does not mean that you cannot restore the data in full, but obviously it should just work.

The logic around the error indicates that one of the index files is read, and this dindex file references a dblock file that is no longer present. Usually, this gets sorted later down where it removes all missing files that are not needed. I think you see a different exception in the end?

That would be awesome!

When it fails, it should give the full name of the dblock file that is missing. If you have the original SQLite database, it should be possible to search that database and see when the file was deleted. This would be in the RemoteOperation table. If you can find the file there, it should be possible to backtrack and see what situation caused the file to be deleted.

I assume that the deletion was intentional from Duplicati as it would otherwise give an error message when listing files prior to running a new backup.

My guess is that this happens during compacting, and perhaps the order of the remote file deletion causes the dindex files to wrongly list a file that will soon be deleted. We do have a bunch of tests around compaction, but this could be something that requires a special setup to be triggered.

As mentioned by @ts678 this was discovered and has been fixed. A new canary build should fix the issue.

1 Like
5

The missing file errors you saw previously (not this instance) might be caused by this:

The issue was discussed here:

Essentially it is caused by an interrupted file upload which leaves the database in an incorrect state. The next backup recovers from the interruption, but it leaves the index file with a broken reference.

2 Likes
6

Hi @ts678, @kenkendk, @Jojo-1000,

Thank you All for your replies.

Indeed, I’ve been working on an old version (2.0.6.3) for quite some time.

I suspect Jojo’s answer is relevant. I have a daily automatic backup running in the background, so it’s well within the realm of possibility that I’ve had interrupted backups in the past (e.g. computer shutdown, network connection lost, etc…).

Unfortunately, I do no longer have the original DB, since I was for quite a while on 2.0.6.3, upgraded directly to 2.0.9.108, and due to the “temporary” column error I downgraded to 2.0.8.1 and decided to rebuild the DB (I figured, perhaps incorrectly, that 2.0.8.1 would not work on the “upgraded” 2.0.9.108 DB).

Now, my current DB was successfully rebuilt on 2.0.8.1.

But, when doing my DRP tests directly from backend storage, I am encountering a MissingFileDetected error. Nevertheless, my latest backup restores just fine (no errors, and when I compare the restore with the source data using winmerge all seems good).

Exact error during the DB repair phase was:
2024-10-26 17:59:54 +09 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b6fe6e2f12bb8478c942f692a6f8287d8.dblock.zip.aes by duplicati-i0fbd36f50cd34f878178316fc62f21e8.dindex.zip.aes, but not found in list, registering a missing remote file

When I check the backend storage:
duplicati-b6fe6e2f12bb8478c942f692a6f8287d8.dblock.zip.aes does not exist
duplicati-i0fbd36f50cd34f878178316fc62f21e8.dindex.zip.aes does exist, modified date was 2024-03-16.

Indeed, back in March, I was running old version 2.0.6.3, which did not include the fix mentioned by @Jojo-1000.

I understand the version I am running now (2.0.8.1) includes the fix to the cause of this problem, but old data on the backend would still be impacted.

Is there a way I can?:

  1. Confirm that none of my valid backups (the ones kept by the retention policy) is impacted by this missing dblock? Hopefully a more efficient way than test-restoring ~100+GB of data from the Cloud 18 times :slight_smile:
  2. How can I “repair” such inconsistencies from my database and backend data?

Thanks in advance !

Gaston

7

The fix for this issue is not applied in any released version, probably because there was no time to review it yet. Fortunately, it should be possible to recover from this manually and it doesn’t lose any data.

You can rename the index file with the missing reference so that it is not recognized (for example duplicati-*** to _duplicati-***). Then a new database recreate should work without errors. If there is missing data, it should be reported during that as well.

If renaming the extra index file doesn’t help, you have a different kind of error, so keep the file for more troubleshooting.

8

Hi @Jojo-1000

Thank you very much.

I spent some time reading how Duplicati works:

Very interesting reading.

If I understand correctly, the purpose of the dindex files is to know which hashes are stored in which dblock files.

I manually decrypted a few dindex files and looked inside of them, and it seems like there is a 1:1 mapping between dindex and dblock files (each dindex file references only one dblock file). Therefore, if I delete the dindex file in the error message, normally it should only impact the dblock file (which anyway is missing). In other words, there is no risk that I break anything other than what is already broken. Is my understanding correct?

Further, let’s say (not the case here, but just for my knowledge) that what was lost is a dindex file, but not its referenced dblock file. In that case, does dupicati automatically rebuild the dindex file from the dblock file? But if so, considering that the file names of dblock and dindex are different, how can duplicati know which dblock (over thousands of files) to download to rebuild the dindex file? I’m guessing it uses the local DB to know which dblocks have dindex file missing?

Sorry for such n00b questions. I find this topic very interesting, so I try to take the opportunity to learn about it :slight_smile:

Cheers,

Gaston

9

From original post of the Issue, here was my guess, and I didn’t read enough to see if it’s wrong:

Regardless, the extra dindex file is harmless until dblock+new dindex from the fix get deleted by a compact. After that, there’s old dindex pointing to a dblock that’s not there. Before that, maybe recreate was happy because it found two dindex, but both pointed to the same dblock, and had the same information, so no harm done. Just a guess.

Specific cases may vary anyway, so conservative path is to not delete files early and regret later.

What I like to see on Recreate is progress bar not getting past 70%. Past 90% can be very slow.

This range is downloading dblocks, and a great view is in About → Show log → Live → Verbose.

If recreate problem came up within the last 30 days (if at default log-retention), database has the history to see a duplicati-b6fe6e2f12bb8478c942f692a6f8287d8.dblock.zip.aes deletion, unfortunately a recreate every 3 months might have missed it. You could try an SQLite viewer on RemoteOperation table Path column to see if it’s seen. DB Browser for SQLite can filter column.

Probably, but I don’t like to talk in certainties when situation is uncertain and being investigated.

I think that’s right IIRC. Maybe Remotevolume table ID and IndexBlockLink table for starter.

This is default case, but it can get out of whack, and it doesn’t seem to have any advance test.

Possibly this is because (I think) dindex files are not technically required, e.g. there’s an option

  --index-file-policy (Enumeration): Determine usage of index files
    The index files are used to limit the need for downloading dblock files when there is no local database present. The more information is recorded in the index files, the faster operations can proceed
    without the database. The tradeoff is that larger index files take up more remote space and which may never be used.
    * values: None, Lookup, Full
    * default value: Full

but it’s annoying because if you lose a dindex file, then recreate the database, then it will be in a forever slow recreate situation because it won’t make a dindex for the dblock, even though it has enough information. I think one time in test, I edited the database to trick it into recreating dindex.

10

Thanks @ts678 for the additional info.

For this particular case, as I downloaded the dindex file, descrypted it and confirmed that it only referenced one dblock file, which was the missing one, I guess I am “safe”.

So, I removed it from the backup storage (kept a copy just in case), and then rebuilt the DB. This time, it rebuilt without any errors.

I also made a direct restore from backup files (the DRP test I mentioned earlier). That also worked flawlessly.

So, that solved the issue! Thanks Everyone for your valuable input.

Best regards,

Gaston

1 Like