Recreate database taking forever

My database wouldn’t repair so I ran recreate (230GB backup).

It seems to be downloading the entire backup (4 days counting on my rubbish internet / approx a 50mb block every 2 mins).

Is this normal behavior?

The database has also only increased in 40mb since the first hour which seems odd.

Presumably this is just temporarily stopping me from backing up, rather than being a potential issue should I need to restore?

Looking at verbose logs it is still running (processing blocklist volume 2950 of 4612), but that’s another 2 days to completion.

The process bar is a bit confusing as it zoomed through 90% in less than an hour and then it started downloading the blocks.

I’m guessing it doesn’t always have to download the blocks so the process bar is simply misleading? (When it decides to do this it would be useful if the block count was displayed in the process bar, with the process amount changed/reset accordingly. A simple ETA would also be helpful / calculated based on how quickly its processed the blocks to date.)

… although slightly concerned a chance my backup has bigger issues?

Sounds like you are using an older version that had a bug in the database recreation process where it sometimes would download all dblock files. (A very lengthy process.)

I recommend aborting the recreation process, installing 2.0.5.1_beta, and then trying the recreation again.

Progress bar looks like will get changed based on the progress within each of the final three sections corresponding to 70%-80%, 80%-90%, 90%-100%, however each phase is calculated after previous phase finishes, and might not run at all. The 90%-100% one is the final desperation attempt to locate whatever’s missing. In Beta before 2.0.5.1, this was thrown off by empty source files (common thing), therefore 2.0.5.1 helps, however if the full message was something more fully stated (code below) as

Pass 3 of 3, processing blocklist volume 2950 of 4612

then you might already on 2.0.5.1. It might have downloaded dblocks from 70%-90%, but passed fast.

Something must have prompted the Repair. Do you recall the symptom, and what Duplicati version was in use at the time (and also now)? 2.0.5.1 is less likely to damage a backup, but can’t Repair everything. There’s a chance it will come up happy when it reaches 100%, but seeing this final pass run is worrying.

Was using a previous version, tried a repair then a recreate but had a power failure mid process.

Since then I updated, with this recreate going already running 2.0.5.1_beta_2020-01-18

In the log I’m getting:

Pass 3 of 3, processing blocklist volume 3164 of 4612

Code doesn’t look like it stops in the middle. If you like, wait until it finishes pass 3, and see if it finds anything wrong. Knowing that (or knowing the original issue that prompted Repair) will help analysis. Frequenlly there’s only a limited amount of loss, and it can be cleaned up by purging the broken files.

Alternatively (e.g. if your downloads are priced by usage, and the old history in backup isn’t valuable), starting over on 2.0.5.1 might be faster. If you prefer to restore old versions first, there are other tools. Depending on storage space and charges, you could do both – keep old backup and start a new one.

Disaster Recovery covers some of what I’m talking about, but it’s your call on what’s important to you.

Sounds like it still has to download some dblocks in your case. Hopefully this number is better than the total number of dblocks you have on the back end, and it will finish soon.

Up to 3990 so should be done tomorrow.

I’m using B2 Backblaze. All it seems to tell me when I log in is that I have 9234 files. For “better” hopefully you mean fewer so I should be okay?

Thanks for the input all.

B2 says you have 9234 dblock files? Or total files?

Total files. I don’t know if there is anyway of seeing/counting just dblock files. I only use it for Duplicati.

Got home and recreate has finally (presumably) finished after a week.The following error was showing at the bottom of the browser:

Recreated database has missing blocks and 1 broken filelists. Consider using “list-broken-files” and “purge-broken-files” to purge broken data from the remote store and the database.

Going into stored log have two messages:

27 Feb 2020 17:20: Failed while executing “Repair” with id: 1
Duplicati.Library.Interface.UserInformationException: Recreated database has missing blocks and 1 broken filelists. Consider using “list-broken-files” and “purge-broken-files” to purge broken data from the remote store and the database.
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run(String path, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal(IFilter filter)
at Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
at Duplicati.Library.Main.Controller.Repair(IFilter filter)
at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

and

27 Feb 2020 17:20: Failed while executing “Backup” with id: 1
Duplicati.Library.Interface.UserInformationException: The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.
at Duplicati.Library.Main.Operation.BackupHandler.d__20.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.b__0(BackupResults result)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
at Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

The second error is somewhat concerning but I ran “list-broken-files” with --full-result which showed 255 files, and then ran "purge-broken-files” which then gave:

Finished!

Uploading file (8.56 MB) …
Deleting file duplicati-20200216T192723Z.dlist.zip.aes …
Return code: 0

It’s now let me run a back-up (in progress), but it appears to be going correctly so far.

Now getting error:

Unexpected difference in fileset version 0: 27/02/2020 18:30:56 (database id: 8), found 99228 entries, but expected 99463

Was that at the end of the backup that seemed to be going correctly so far, or was it a later backup?

The missing blocks exception is consistent with the long search attempting to find all needed blocks.

Seemingly the backup is damaged, more likely on the older Duplicati before 2.0.5.1 which is better…

I suppose you can run list-broken-files again to see if it lists something again, but prior fix didn’t stick.

Sometimes the bad version can be deleted, but sometimes the failure then moves to a different one.

Version 0 is the latest version, and the easiest way to delete it is to use Commandline, pick Delete, change Commandline arguments into --version=0, then Run "delete" command now at bottom.

Then the question is whether this will stick, or the next backup will fail either at its start or at its end.

What was the problem leading to the opening “My database wouldn’t repair”? Possibly it’s still there.
Fixing damaged backups can be very difficult, as evidenced here. Is old history in this one valuable? Sometimes damaged backups can still be kept for restores. We’ll see if this one will backup again…

I had this problem some days ago, and just started my backup from scratch, losing all retention files.
It’s a bit frustrating that some backups can’t be repaired.

Backing up the local SQL database after a successful backup could avoid this?

Not guaranteed, especially if one is talking about a block that went missing from the destination.
Before 2.0.5.1, there was a bug found/fixed in compact that could drop a block from the backup.

Backing up the database might help in some cases if database is damaged, but remote is OK…
Generally that should be handled fine by a Recreate. Not in this case, due to backup damage…

1 Like

Tried running again list/purge-broken-files again (it found some of the same files). Rerunning backup.

Is there a way of testing for how badly you’re affected by this missing blocks bug? And, if the files are still present locally, recovering?

From experiment and a look at code I don’t know well, it looks like “Unexpected difference in fileset” is one of the self-checks that this can error, however even though the numbers discrepancy is a possible metric, what’s missing is whether other versions are affected. The consistency check stops on the first problem. Testing delete of the affected version is needed, unless you want to run a lot of manual SQL.

“Found inconsistency in the following files while validating database:” is near the first check, and looks like it ought to detect problems with a missing block that doesn’t wind up with a completely missing file. One forum report that I sampled (there aren’t a lot) had a file over 6GB that was short by 102400 bytes which happens to be exactly one default –blocksize, so possibly was this problem before it was known.

VerifyConsistency code runs before and after every backup, so if it’s not complaining you might be OK.

I tested one of my own old backups with “Unexpected difference…”, and list-broken-files and –dry-run repair didn’t flag anything. The TEST command gave “Unexpected difference …” before getting going.

1 Like

Received error:

Error: “Detected non-empty blocksets with no associated blocks!”

Tried “list-broken-files” which showed 39 files (same files as before), then tried “purge-broken-files” which gave following error:

Finished!

ErrorID: CannotPurgeWithOrphans
Unable to start the purge process as there are 90 orphan file(s)
Return code: 100

Watching this issue… I’ve been going without backups since Duplicati blew up in December, 2.0.5.1 didn’t fix my problems (also a Backblaze B2 user), and I’m hoping that the fundamental issue is addressed before I wind up having to find an alternative backup solution (probably one of the paid ones like MSP360).

The fundamental issue is that network backups by their nature have to journal all their actions and have airtight rollback / rollforward mechanisms to handle inevitable dropouts in network connectivity. It has to assume that any non-immutable remote file can be corrupted at any time.

This seems a little off topic for “Recreate database taking forever”, but here’s a reply to the comment. Getting into problem specifics belongs in a new topic or GitHub issue, preferably with steps to cause.

2.0.5.1 is not much better at fixing problems that already exist (except DB Recreate is usually faster),
but it should be better at staying intact through ordinary operations.Prior to this, there were logic bugs which could cause broken backups, but some bugs almost surely remain. Good bug reports help a lot.
Hard stops are being worked on now, but I don’t know if SQLite level corruption is fixable by Duplicati.

If the entire remote backup gets trashed, there’s not much that can be done. Duplicati does verify that names and sizes are as expected, and it sample files by actual downloads and verification of file hash. Sample size can be increased as desired, all the way up to everything, every backup, but that will hurt.
md5 & sha1 inclusion in verification json #2189 is an enhancement request to use cloud hash facilities.

I’m not sure they’re airtight, and it’s worse than that because backends can do all sorts of weird things.
Recover automatically from interrupted backups #1243 shows states that a remote file moves through.

Especially with B2 free uploads, why not just start a backup on 2.0.5.1 and see if it holds up, and detail problems that seem due to network or remote? Reproducible issues are ideal, as those can be chased.

1 Like