Can I use an old version of the database?

Support
1

I noticed about a week ago that my backup had been failing for the past month or so due to a file size mismatch on a single file. I followed the recommendation to restore but the database restore failed (I can’t remember the error message that was given). I went to recreate the database, but the process has been taking an unusually long time (seemed to be at 70% for about a week, and then Windows Update decided it was time to reboot)

I have a version of the database from the last successful backup. Would I be able to use that in some way to ease the repair/recreate process?

Alternatively, could I drop that old database in and rename the offending file to something else so Duplicati stops hiccuping on it?

edit: I believe the old database version is from Duplicati 2.1.0.5, which was also where the last successful backup was from.

My main concern here is that at the current state of the database recreate, it’ll process a 50.11MiB blocklist file (the last message I’ve gotten on that was Pass 1 of 3, processing blocklist volume 12 of 1923…), do about 300GB or so of writes to my temp directory (note, my pagefile exists on the same drive so some of those writes may have been that), and a couple hours later start the next block. At this rate I should be done recreating the database in about half a year, which feels very slow, and I don’t want to keep my computer constantly running for that long. To say nothing about apparently needing multiple bytes per bit of the blockfile in order to process it.

Duplicati Version: 2.2.0.3_stable_2026-01-26

Source: 6.69TiB

Destination: 8.8TiB

2

Update to my situation: I did manage to get the old database working. I’m currently trying to do a list-broken-files (I’m using spare space on my NAS for the tempdir so it’s slow going, since I really don’t feel like chewing through an SSD’s lifespan for this) and hopefully that will identify the file(s) that it’s taking issue with.

The exact error I’m getting is (note, this was on a purge but I get the same thing no matter what I run)

The operation PurgeFiles has failed => Found inconsistency in the following files while validating database:
F:\Videos\Twitch Streams\2020-08-11 20-12-41.mkv, actual size 2056195055, dbsize 2056092655, blocksetid: 2435466
. Run repair to fix it.

Strangely, if I run repair I get the exact same message. From above, I know the exact filename, and I know it’s one that hasn’t changed at all since I started this backup set, so history shouldn’t be a problem. If I try to run the purge command on that file I get the same DatabaseInconsistency error. Is there a way to bypass that check and remove the that one file, then let it get re-added on the next backup run?

3

There is no tool as such for handling this.

What the check does is look at the size of the file, and then the sum of the size of blocks required to recreate the file, and these two numbers do not match for that particular file.

Looking at the numbers it is exactly one block that is missing somewhere: 2056195055-2056092655 =102400.

If it is possible, I would like to get a bugreport database so I can see the damage. If you can also create a bugreport database from the “new” version of the database that did not have the issue, I can try to compare them and see if I can figure out where the block was dropped.

If the goal is to simply get the database running again, you can edit the SQLite database.
You need to find the FileID for the path and blocksetid 2435466.
Then you can delete all occurences of that file in FilesetEntry and FileLookup.

Once that is done, the next backup will see a “new” file and back it up like nothing has happened. Be careful if you try this as modifying the database manually is always risky.

Note that this does not fix the data you already have, so if you need to restore without the database from an older file, you will likely see the same issue again.

You can make Duplicati recreate the dlist files if you remove/rename them on the storage and then run repair, but be careful because repair can also delete remote files it does not like.

4

All right, I’ll try that tonight. list-broken-files is still going, and I did get a hit:

Starting - ExecuteReader: SELECT "ID", "Timestamp" FROM "Fileset" ORDER BY "Timestamp" DESC
ExecuteReader: SELECT "ID", "Timestamp" FROM "Fileset" ORDER BY "Timestamp" DESC took 0:00:00:00.000
Fileset 38      : 18/Feb/2023 13:00:00  (1 match(es))
        F:\Videos\Twitch Streams\2020-08-11 20-12-41.mkv (1.91 GiB)

Is Fileset here a prior bacakup? The web interface reports 40 backups, if it’s counting backwards with 0 being the oldest, I could also drop the last couple of backups (this would obviously be a last resort because I’m fully aware that’s a non-reversible process)

My database is sitting at about 47GiB, so I can create a debug version but unless it massively reduces the filesize, uploading it somewhere might be an issue.

5

A Fileset in the database is called a snapshot or backup entry elsewhere. And the UI shows the most-recent as version 0.

Yeah, that does sound like a hassle to get uploaded and downloaded.

6

FWIW SwissTransfer file size limit is 50 GB, but I assume it will take awhile.

7

Ran a dry-run of purge-broken-files and it did successfully flag that one file for deletion, and found the associated dlist with it. (it isn’t fully done yet but so far it’s looking promising)

I think something’s off with the output though, since this popped up (I had the logging level to profiling to see what exactly the dry run would do):

Starting - ExecuteScalarInt64: SELECT COUNT(*) FROM "FilesetEntry" WHERE "FilesetID" = 158
ExecuteScalarInt64: SELECT COUNT(*) FROM "FilesetEntry" WHERE "FilesetID" = 158 took 0:00:00:00.037
Starting - ExecuteScalarInt64: INSERT INTO "Remotevolume" ("OperationID", "Name", "Type", "State", "Size", "VerificationCount", "DeleteGraceTime", "ArchiveTime") VALUES (1310, "duplicati-20230325T200001Z.dlist.zip.aes", "Files", "Temporary", -1, 0, 0, 0); SELECT last_insert_rowid();
ExecuteScalarInt64: INSERT INTO "Remotevolume" ("OperationID", "Name", "Type", "State", "Size", "VerificationCount", "DeleteGraceTime", "ArchiveTime") VALUES (1310, "duplicati-20230325T200001Z.dlist.zip.aes", "Files", "Temporary", -1, 0, 0, 0); SELECT last_insert_rowid(); took 0:00:00:00.000
Starting - ExecuteScalarInt64: INSERT INTO "Fileset" ("OperationID", "Timestamp", "VolumeID", "IsFullBackup") VALUES (1310, 1679774401, 437919, 0); SELECT last_insert_rowid();
ExecuteScalarInt64: INSERT INTO "Fileset" ("OperationID", "Timestamp", "VolumeID", "IsFullBackup") VALUES (1310, 1679774401, 437919, 0); SELECT last_insert_rowid(); took 0:00:00:00.000
Replacing fileset duplicati-20230325T200000Z.dlist.zip.aes with duplicati-20230325T200001Z.dlist.zip.aes which has with 1 fewer file(s) (10,857.98 TiB reduction)
Starting - ExecuteReader: SELECT "B"."Path", "C"."Length" FROM "TempDeletedFilesTable-054473B73C1C9242A975704B2948A4B8" A, "File" B, "Blockset" C WHERE "A"."FileID" = "B"."ID" AND "B"."BlocksetID" = "C"."ID"
ExecuteReader: SELECT "B"."Path", "C"."Length" FROM "TempDeletedFilesTable-054473B73C1C9242A975704B2948A4B8" A, "File" B, "Blockset" C WHERE "A"."FileID" = "B"."ID" AND "B"."BlocksetID" = "C"."ID"  took 0:00:00:00.000
  Purging file F:\Videos\Twitch Streams\2020-08-11 20-12-41.mkv (1.91 GiB)
  Purging file F:\Videos\Twitch Streams\2020-08-11 20-12-41.mkv (1.91 GiB)
[Dryrun]:   Purging file F:\Videos\Twitch Streams\2020-08-11 20-12-41.mkv (1.91 GiB)
[Dryrun]: Would write files to remote storage
Writing files to remote storage
[Dryrun]: Would upload file duplicati-20230325T200001Z.dlist.zip.aes (323 bytes) and delete file duplicati-20230325T200000Z.dlist.zip.aes, removing 1 files

My NAS capacity is 36TiB or so, so I don’t know where Duplicati expects to store or reduce this 10PiB or so of data from. Did a signed value get shifted to unsigned at some point? The file in question is 1.91GiB, so I’m not sure where this other value is coming from. It also looks like the dlist would be reduced to 323 bytes from 423MiB.

8

What sort of file is this, from its file name. I guess dblock, but please confirm.

“10,857.98 TiB reduction” is discussed in below. Its fix seems in today’s Beta.
Because it seems to be a bad report, jumping into Beta might not be worth it.

I’m not sure if it relates to the too-small-looking dlist. Might be a different bug.
Developer might have an opinion. I see other oddly small ones in the reports.
Possibly it’s a benign mis-report, but I have no other opinion on it beyond this.

9

Sorry I wasn’t clear. When I said “one file” I’d meant there was one broken file to purge. But it’s going through all 39 filesets to purge that file.

Question on this, in part because I’m concerned about write cycles on my SSD (though granted I did specifically buy it to be a scratch drive for I/O intensive stuff): Right now this was a dryrun, but would I have to go through 39 filesets’ worth of data again when I do the actual purge-broken-files? Or would the broken file get removed on the first fileset and the remaining ones proceed much quicker?

edit: Here’s a link to the dry-run log (so far, it’s still “working” on the remaining filesets)

10

provides background on concepts and terminology, at least as I understand.
Duplicati 2 (unlike Duplicati 1) uses standalone versions, plus deduplication.
If you purge a broken file, however many filesets/dlist it is in must reflect that.

The dlist is a full list of file paths, plus enough info to figure out the block use.
DB Fileset table is like dlist backup versions. FilesetEntry table names blocks.

All of this is SQL queries I think. The dlist is small compared to whole backup.
SQLite can use temporary files if memory runs out, but I think it’s more reads.
Similarly, it writes DB some as it goes, but that’s going to happen for anything.

The difference between dry-run and not in terms of remote writes is dlist files.
Some dblock file blocks will also turn into partially or completely now-useless.
This may trigger compact, which will delete or repackage, just like usual case.

11

I made a bugreport database (and I’ll make another one after running the purge-broken-files)
It seems on the web interface it’ll only create the database if tasks are not paused, but I’d like to make the second bugreport database before running another backup task. Is there a command line equivalent I can run?

Here’s the one I made before running purge-broken-files for real (now with the actual link posted, sorry about that).

12 
Duplicati.CommandLine help create-report

Usage: Duplicati.CommandLine.exe create-report <storage-URL> <output-file> [<options>]

  Analyses the backup and prepares a report with anonymous information. This report can be sent to the developers for a better analysis in case something went wrong.

but you should give an explicit --dbpath option pointing to the job database.
Without it, true (not GUI) command line may assign you a new database path.

13

I did run into that issue at first with the command line stuff. My correct DB path is in the dbconfig.json file, but I’ll specify it to be 100% sure.

14

Three days and about 17TB of host writes to my SSD later, the dataset is purged of the rogue file and Duplicati is happily backing up.

In short: Yes the old version of the database worked, since I had no additional backup history when trying to drop it back in. I’m not sure what would have happened if the local database had been behind what the remote files were expecting.

purge-broken-files helped solve the database inconsitency, but it was really heavy on writes. I’m not sure if it had to download the entire dataset or if the amount of cached data is kept low and it was constantly cycling the same blocks in and out again.

Here’s the debug database immediately after the purge but before the next backup.

(@kenkendk let me know when you’ve grabbed these so I can free up OneDrive space, and sorry about the size)