Duplicati Database Recreate running for a week

gpatel-fr · August 16, 2023, 10:55pm

not sure it needs to be so long.

See my branch that I updated just now to preview a solution to this problem. It still need more thought and tests but so far it has withstood what I have thrown at it.
The basic idea comes from this thread. I have always suspected that this kind of problem could come from a backend bug or even a basic misstep from users, massively deleting index files. However this thread shows that actually the problem can occur with more index files that data files. So I have searched instead in the direction ‘can an invalid index file create this situation’. And bingo, I have found a way to generate this problem. Now I don’t know at all how Duplicati could do that. Maybe it’s not the same kind of invalid index file in this thread. Possibly this is a race condition, probably in compact (because it seems just the more dangerous operation, especially if it crashes at the wrong time).
IMO it does not matter. Whatever the reason, if there is a way to make this process faster it is a bug fix in its own right.

So the idea: basically the main reason this stuff is slow is that there is a massively expensive query running for every block file. I have searched to accelerate it (it is the first part of my branch). This was moderately successful, but it was a drop in the ocean. Reducing a query from 2’ to 1’50 does not count for much when it is run 20000 times.

So I have considered, given the new information that a single invalid index file can generate a massive test of all block files, that it is not very appropriate to run it for every block file. Indeed, when one index file is invalid out of 20000, we have 19999 valid block files already correctly handled by the previous index files scanning. So it makes sense to try to check if the block file handling has not produced any new information and if yes, skip the query that is supposed to update the tables based on the new information. The fastest query is the one that does not run. This is the most important change in the update to my branch today (there are other minor changes also but are just details in comparison).

here is the relevant part:

github.com

duplicati/duplicati/blob/32fcab927685d11c760c1260962eac4894977511/Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs#L510


      
                                                  if (restoredb.AddTempBlockListHash(blocklisthash, rd.ReadBlocklist(blocklisthash, hashsize), tr)) {
                                                      anyChange = true;
                                                  }
                                              }
          

                                              // Update tables if necessary (if no block or hash have been changed by a data volume
          				    // there is no need to run expensive queries - most data volumes have been
          				    // managed successfully by correct index volumes), so we know if we are done
          				    // if any change, add to the block and blocksetentry tables the references found in
                                              // the block lists of the volume saved in the temp blocklisthash table by AddTempBLockListHash
                                              if (anyChange) {
                                                  if (i == 2) {
                                                      Logging.Log.WriteWarningMessage(LOGTAG, "UpdatingTables", null, "Unexpected changes caused by block {0}", sf.Name);
                                                  }
                                                  if (expRecreateDb)
                                                      restoredb.AddBlockAndBlockSetEntryFromTemp(hashsize, m_options.Blocksize, tr, false);
                                                  else
                                                      restoredb.FindMissingBlocklistHashes(hashsize, m_options.Blocksize, tr);

                                              }
          
                                              using (new Logging.Timer(LOGTAG, "CommitRestoredBlocklist", "CommitRestoredBlocklist"))

for your (and @ts678) viewing and comment (I have serious concerns about changing this part of the code…)

me neither, I would massively prefer to avoid it.

in my rebuild branch, I have actually added an option to force the dblock handling, since I have seen a case of broken index file (I have generated 2 different cases) where the badness was not detected at first (so no dblock handling), but at the end, so the rebuild was a failure. Forcing dblock handling can fix it. Execution time is similar to a full backend test (that is, slow as in hours, not slow as in days).

I could create a build of the branch for you if you can take the time to test it on the original problem backend, it would be much appreciated. Thanks anyway for posting the information you did, it was very instructive.