Recreating database logic/understanding/issue/slow

Sample log output shows it downloading a dblock, then doing a sequence of SELECT with no other SQL:

SELECT “VolumeID” FROM “Block” WHERE “Hash” = “04++WDfbojSWsXsKjPSmxdrOgKkD9D8W4XZaow8nhsk=” AND “Size” = 102400

A sequence of about 50 per dblock is due to default dblock size of 50MB and dedup block size of 100KB.

Reversing from SQL suggests this is m_findHashBlockCommand being run in UpdateBlock() from code:

and (speculating further), an empty source file in the backup can cause a reference in the dlist without actually having an empty entry in the dblock (which implies it’s not in the dindex for a dblock file either).

Jumping way ahead, I speculate that this is due to incorrectly treating the empty file as loop terminator:

which means you wind up with no empty block put out. You do wind up with an empty blockset though.

This is reproducible. Just back up an empty source file, watch live log at Information, and do Recreate.

Lacking a full log, the way we’d have to look at this case would be to determine if we dare copy the DB currently being recreated. I’d sure hate to have recreate die due to a locking error – Windows generally allows me to leave DB Browser for SQLite read-only on a database until it needs to be deleted or such, implying that a copy here is probably also safe, however I’ll leave to @f4253a38b0 to make that call…

Finding the database-in-progress would probably mean looking in the temporary files, guessing at files, copying, and test-opening with a database browser. The suspect is a Blockset of Length 0, and further forecast of a wasted dblock loop is Hash not being in Block table at all, or being there with VolumeID -1.

I was creating some materials for a GitHub issue, but maybe I’ll at least see what I can get started here.

lengthtest - length 0 backup delete repair.zip (43.8 KB) shows the not-typical situation from empty file

lengthtest - length 1 backup delete repair.zip (45.2 KB) shows a more typical situation from 1-byte file

The test was to do the initial backup, bug report, recreate (File Explorer delete + GUI Repair), bug report. There are separate Profiling (plus --profile-all-database-queries) logs from the backup and the repair too.
A final bonus is the actual source file, plus sqldiff output showing DB differences after Recreate such as:

INSERT INTO Block(ID,Hash,Size,VolumeID) VALUES(2,‘47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=’,0,-1);
INSERT INTO BlocksetEntry(BlocksetID,“Index”,BlockID) VALUES(1,0,2);

which is how Recreate changes the DB style of the problem from missing BlocksetEntry to -1 VolumeID. Either way, empty file has made a break in the usual chain from File to BlocksetEntry to Block to Volume.