My experience with Database Rebuild / Recreate


#1

Let me start by saying this is all academic. The backups in question are not critical and the issues mentioned are all based around TESTS I’m running to look into the long database recreate times people are seeing - I actually have the original pre-error database backed up so can get back to backing up any time I want. :slight_smile:


As mentioned in "missing files" error after switching to custom retention policy, I ran into a Found 20 files that are missing from the remote storage situation with my backup that required a Repair.

I did the repair, which took all of 42 minutes for my 853MB sqlite database, and my backups started working as expected. Yay!

I then decided my life was too boring so I renamed my (now repaired) database and decided to try a Database Recreate. As mentioned in How long should a repair of a database take? 4 days, 5 hours, and 37 minutes later my recreate was all done. Yay! Sort of…

Here’s what the Messages looked like: :+1:

Messages: [
    Rebuild database started, downloading 17 filelists,
    Filelists restored, downloading 4429 index files,
    Processing required 55 blocklist volumes,
    Processing all of the 2392 volumes for blocklists,
    Recreate completed, verifying the database consistency,
    Recreate completed, and consistency checks completed, marking database as complete
]

Here’s what the Warnings looked like: :thinking:

Warnings: [
    The 7z compression module has known issues and should only be used for experimental purposes,
    Mismatching number of blocklist hashes detected on blockset 11938. Expected 0 blocklist hashes, but found 1,
    Failed to process file: duplicati-20180309T154050Z.dlist.7z.aes => Found invalid data while decoding.,
    Failed to process file: duplicati-20180316T222211Z.dlist.7z.aes => Invalid JSON, expected EndObject, but found PropertyName, "metaha9b8dYlc3d\Me\AppData\Local\Plex Media Server8730e\3\620a7e1c5904ae8d6b0747ae4863fa03ea1ce6b.bunGf05Ke75a3c62n74combined\v8d3h" while reading entry C:\Users\Me\AppData\Local\Plex Media Server\Metadata\TV Shows\4\4e95fbe50e66b2d78fa0d06a9cf9689e6a55787.bundle\Contents\_combined\art\com.plexapp.agents.thetvdb_83249fa2d0df578410a5d5e5ddcaa207683e5c0d,
    Failed to process file: duplicati-20180320T042324Z.dlist.7z.aes => Invalid JSON, expected property "metasize", but got PropertyName, metahash
]

And no errors were listed.

Yet at the next scheduled run I got two errors:

  • SharpCompress.Compressors.LZMA.DataErrorException: Data Error
  • Found 20 files that are missing from the remote storage, please run repair

“But wait,” you’re probably saying to yourself… “that missing files error is what started this whole thing, isn’t it?” to which I answer “yep”.

So my question is how can the first thing to happen after a database RECREATE be that files are missing? (I did verify that they do not exist on the destination, a local SFTP server.)

Since all 20 reported missing files are dindex files I’m trying another Repair (it should only take about 45 min, right?)…


Check rebuild DB operation progress
Is database retention delay normal? Duplicati - 2.0.3.9_canary_2018-06-30
Very slow database rebuild
#2

For the record, the 2nd repair seems to have done the trick and my simulated database failure has recovered - though via a less than optimal route.


#3

Yeah, that is not nearly good enough.

That one I am not sure about, the others are likely a result of using 7z compression.

It happens because the recreate builds the database with all the information it can find. For some reason the missing files showed up in one or more of the existing files. You should see some messages like “registering missing volume” when that happens.

After the recreate it knows that the files should have been there, and then it complains.