Database disk image is malformed

Support
22

Hello, folks,

I know it’s a few months later but I just had this issue myself and I think I found a solution:

@jonmikeiv, Maybe the person with a lot of knowledge on the DB can chime in on if this method is inappropriate . But, here is what I did.

First, stop services, as a precaution.

Now, make a backup copy of the {fooo}.sqlite file.

Next, load the bad DB in your favourite SQLite editor. I use DBeaver, but the exact tool is un important.

Then I did:

pragma integrity_check;

/* RESULT: (takes forever)
	integrity_check                             |
	--------------------------------------------|
	row 34538 missing from index FilePath       |
	row 56151 missing from index FilePath       |
	row 56162 missing from index FilePath       |
	row 172197 missing from index BlockHashSize |
	row 220677 missing from index BlockHashSize |
	wrong # of entries in index BlockHashSize   |
 */

As we can see, the errors solely affected indexes. So, I investigated the DDL of the tables and recovered that the indexes were created like this:

CREATE UNIQUE INDEX "FilePath" ON "File" ("Path", "BlocksetID", "MetadataID");
CREATE UNIQUE INDEX "BlockHashSize" ON "Block" ("Hash", "Size");

We now have enough information to remove the corrupt indexes and recreate them.

drop index "FilePath";
drop index "BlockHashSize";

CREATE UNIQUE INDEX "FilePath" ON "File" ("Path", "BlocksetID", "MetadataID");
CREATE UNIQUE INDEX "BlockHashSize" ON "Block" ("Hash", "Size");

It was fast for me, but index creation could be quite slow.

pragma integrity_check;

/* SUCCESS!
	integrity_check |
	----------------|
	ok              |*/

Then, all I did was copy the db file back over the original and restart services.

I now am able to complete the backup without error:

vivaldi_2018-03-11_21-12-53|513x205

When the error was occuring, the backup halted.

For me, the problem started at about the time that an upgrade didn’t succeed.

So, while I’m not super familiar with SQLite, it seems that at least in cases where this error affects indexes, manual intervention can help.

Though, the obvious question is ‘why did this happen?’ (I have preserved the bad database in case devs !have questions.)

4 Likes
23

And a good question. This should not happen, but it is not something that Duplicati can control. It happens somewhere inside the SQLite library.

24

Perhaps we can catch the error and recreate the indexes in Duplicati in this edge case?

I’m assuming it’s in the job specific DB, which means we just have to make sure we’re not using it before attempting the repair.

We could also write it into the repair method and then make the error trigger a “Please run repair” message?

25

I can say that I had this problem as well and it was the index BlockHashSize. I followed the instructions to recreate the index and everything is fine now. If Duplicati can recognize this error and recreate the indicies, that would be great. It would be better to know the root cause, but I didn’t see anything particular with mine as well.

26

At random* one of my regular jobs started complaining about “disk image malformed”, pulled up DBeaver and got the following when running an integrity check.

(slightly edited result)
integrity_check
Page 51630: free space corruption|
row 696150 missing from index FilesetentryFileIdIndex |
row 696151 missing from index FilesetentryFileIdIndex |
row 1559442 missing from index FilesetentryFileIdIndex |
row 2422734 missing from index FilesetentryFileIdIndex |
row 3286028 missing from index FilesetentryFileIdIndex |
row 5012620 missing from index FilesetentryFileIdIndex |
row 5875916 missing from index FilesetentryFileIdIndex |
row 6739205 missing from index FilesetentryFileIdIndex |
row 7602503 missing from index FilesetentryFileIdIndex |
row 8457470 missing from index FilesetentryFileIdIndex |
row 9316511 missing from index FilesetentryFileIdIndex |
row 10175552 missing from index FilesetentryFileIdIndex |
row 10809664 missing from index FilesetentryFileIdIndex |
row 22460111 missing from index FilesetentryFileIdIndex |
row 23397674 missing from index FilesetentryFileIdIndex |
row 24334312 missing from index FilesetentryFileIdIndex |
row 25270700 missing from index FilesetentryFileIdIndex |
row 26223263 missing from index FilesetentryFileIdIndex |

* The only thing I can think of, I may have put my computer to sleep while that backup job was running.

I was about to drop index "FilesetentryFileIdIndex" but then realized I have no clue how to recreate it. I paused for a moment, then dropped the index anyways… I then reran the integrity check, which it passed and Duplicati just finished running that job.

27

I don’t use DBeaver, but DB Browser for SQLite shows the index’s creation SQL. This one looks like

CREATE INDEX “FilesetentryFileIdIndex” on “FilesetEntry” (“FileID”)

Also see

How To Corrupt An SQLite Database File gives some possible ways. I sleep my Windows on Duplicati randomly (an effort downgrade from sleeping on it on purpose…), though different systems likely differ.