My experience with Database Rebuild / Recreate

Developer
1

Let me start by saying this is all academic. The backups in question are not critical and the issues mentioned are all based around TESTS I’m running to look into the long database recreate times people are seeing - I actually have the original pre-error database backed up so can get back to backing up any time I want. :slight_smile:

As mentioned in "missing files" error after switching to custom retention policy, I ran into a Found 20 files that are missing from the remote storage situation with my backup that required a Repair.

I did the repair, which took all of 42 minutes for my 853MB sqlite database, and my backups started working as expected. Yay!

I then decided my life was too boring so I renamed my (now repaired) database and decided to try a Database Recreate. As mentioned in How long should a repair of a database take? 4 days, 5 hours, and 37 minutes later my recreate was all done. Yay! Sort of…

Here’s what the Messages looked like: :+1:

Messages: [
    Rebuild database started, downloading 17 filelists,
    Filelists restored, downloading 4429 index files,
    Processing required 55 blocklist volumes,
    Processing all of the 2392 volumes for blocklists,
    Recreate completed, verifying the database consistency,
    Recreate completed, and consistency checks completed, marking database as complete
]

Here’s what the Warnings looked like: :thinking:

Warnings: [
    The 7z compression module has known issues and should only be used for experimental purposes,
    Mismatching number of blocklist hashes detected on blockset 11938. Expected 0 blocklist hashes, but found 1,
    Failed to process file: duplicati-20180309T154050Z.dlist.7z.aes => Found invalid data while decoding.,
    Failed to process file: duplicati-20180316T222211Z.dlist.7z.aes => Invalid JSON, expected EndObject, but found PropertyName, "metaha9b8dYlc3d\Me\AppData\Local\Plex Media Server8730e\3\620a7e1c5904ae8d6b0747ae4863fa03ea1ce6b.bunGf05Ke75a3c62n74combined\v8d3h" while reading entry C:\Users\Me\AppData\Local\Plex Media Server\Metadata\TV Shows\4\4e95fbe50e66b2d78fa0d06a9cf9689e6a55787.bundle\Contents\_combined\art\com.plexapp.agents.thetvdb_83249fa2d0df578410a5d5e5ddcaa207683e5c0d,
    Failed to process file: duplicati-20180320T042324Z.dlist.7z.aes => Invalid JSON, expected property "metasize", but got PropertyName, metahash
]

And no errors were listed.

Yet at the next scheduled run I got two errors:

  • SharpCompress.Compressors.LZMA.DataErrorException: Data Error
  • Found 20 files that are missing from the remote storage, please run repair

“But wait,” you’re probably saying to yourself… “that missing files error is what started this whole thing, isn’t it?” to which I answer “yep”.

So my question is how can the first thing to happen after a database RECREATE be that files are missing? (I did verify that they do not exist on the destination, a local SFTP server.)

Since all 20 reported missing files are dindex files I’m trying another Repair (it should only take about 45 min, right?)…

1 Like
2

For the record, the 2nd repair seems to have done the trick and my simulated database failure has recovered - though via a less than optimal route.

3

Yeah, that is not nearly good enough.

That one I am not sure about, the others are likely a result of using 7z compression.

It happens because the recreate builds the database with all the information it can find. For some reason the missing files showed up in one or more of the existing files. You should see some messages like “registering missing volume” when that happens.

After the recreate it knows that the files should have been there, and then it complains.

4

OK - I have a recreate going on 24 hours now. With 7479 volumes. So I can basically expect this to take two weeks?

5

It’s been a while since I tested but I recall it being non-linear. Sometimes my estimate were off by hours, other times weeks or days. I never bothered to figure out what steps caused the less accurate predictions.

That being said, it’s possible it could take that long - though unlikely unless you have a very high volume size and/or slow connection to your destination AND some bad dindex files (forcing downloading of the large dblock files to get the missing info).

It’s more likely that the number and length of file paths in your backup (rather than number of volumes slows) down the rebuild time.

6

I went with deleting and recreating. Looks like that will finish today, anyway.

7

I have a sizable backup and it has been recreating my database for 14 days now. It looked like it was almost done about 10 days ago. I’m doing this as a test, my database was 11GB and I renamed it. It is currently 3.8GB. I have no hope of it actually finishing anytime this month.

2 Likes
8

So it seems that you’ve been having similar experiences than I’ve had.

Yet, I assume that in my case the situation is much worse than just time spent rebuilding the database. My observation is that there are I/O patterns which probably are making this process around 20 times slower than it would be without those adverse patterns at least in my case.

9

My database finished recreation last week with an error saying there were broken files on the destination. I ran the purge-broken-files which took 2 days to delete 4GB, then it said the database needed a repair which immediately fails. I’m switching to duplicacy for now as this is not worth the effort.

10

Well, I tried the recovery options and none of those even worked. Wasted a few days running tests which indicate a problem, but then there’s no other way to recover than full restart, or probably I could have deleted some more files (?) manually, to remedy the situation, but I’m not going to play with that stuff and waste another week.

11

Sorry to hear you had such difficulties with Duplicati. It’s been said before, but it’s a great tool when it works but for the small percentage of people where it doesn’t it can be a pain.

Duplicacy seems like good tool as well, just with a different design goal - I hope it works out for you!