Hi everyone,
I’ve been without a functional backup since April, dealing with a cascading failure on a ~2TB backup set that started after a recent update.
I know from my recovery post last year that the default advice for major database issues is often “just start a new backup.” However, starting fresh means re-uploading (and losing) over 1TB of data with a second copy of a backup, which simply isn’t a practical option for me right now. I need to salvage this existing backup set, and I’m hoping to find out the safest next step from the current repair/recreate state.
The Environment
Normal Backup Setup:
- Source: Windows 11, ASUS ROG Flow X13 (16 GB RAM) gaming laptop running the native Windows Duplicati client.
- Destination: Raspberry Pi 5 (16GB RAM) acting purely as an SSH host, storing the data on an attached 4 TB ext4 disk.
- Backup Size: ~2 TB, ~83,301 remote files, 341 filesets / versions.
Recovery Setup:
To isolate the issue and free up my laptop for work, I eventually moved the database recreate/repair attempts directly onto the Pi 5. These recovery attempts are running in a dedicated, isolated Duplicati Docker stack, testing copied databases with stable/beta images over the last few weeks.
The Timeline & Symptoms
1. The Initial Failure: April
On April 28th, a normal backup from my Windows PC failed with:
Could not load file or assembly 'Duplicati.Library.Interface, Version=2.3.0.1'
This seemed like a partial update or version mismatch on the Windows side.
2. Repair Attempts & Memory Blow-up
I attempted a database repair. For speed, I plugged the 4TB drive directly into a spare ThinkPad, but the repair repeatedly black-screened/crashed the machine. I verified this wasn’t just the ThinkPad by testing it on my main ROG Flow, which suffered the same issue.
Checking the logs, Duplicati was getting stuck on a massive SQLite query over FilesetEntry / File, requesting 60+ GB of virtual memory.
Workaround: I forced SQLite temp storage to disk using:
CUSTOMSQLITEOPTIONS_DUPLICATI=temp_store=FILE
The repair then completed without maxing out RAM.
Note for devs: this may be worth looking into as large-database SQLite temp-storage/resource behaviour, but it is not the main thing I need help with in this post.
3. The Recreate Phase
After the successful repair, the backup failed again stating:
Found 83301 remote files that are not recorded in local storage.
Since this is basically the entire backend, I treated the local DB as stale and moved to a full database recreate.
To do this without tying up my laptop, I moved the 4TB drive back to the Pi and set up the Docker test environment mentioned above. The recreate finished, but final validation reported 36 broken filelists, mostly spanning March 17 to April 20.
4. Purge Dry-Run Issues
I am currently testing purge-broken-files via the Docker test stack using copied DBs to be safe.
When I run:
purge-broken-files --dry-run
it successfully simulates the purge of all 36 broken filelists, but the operation still ends with a DatabaseInconsistency error.
The current stable-image dry run processed all 36/36 broken filesets, then failed final validation with examples like:
D:\MediaMonkey Backups\MM_20260314171509.zip, actual size 43733978, dbsize 26956762, blocksetid: 4049637
D:\Mixed In Key\10.0\AudioAnalysisCache3\1ed128c30a191e73861931fad9ebb3980.wfd, actual size 145673, dbsize 0, blocksetid: 4049643
D:\Mixed In Key\10.0\AudioAnalysisCache3\1ed128c30a191e73861931fad9ebb3981.wfd, actual size 145627, dbsize 0, blocksetid: 4049645
D:\Mixed In Key\10.0\AudioAnalysisCache3\3d12a17055aecb6410f3e5b747af02761.wfd, actual size 161481, dbsize 0, blocksetid: 4049649
D:\Mixed In Key\10.0\AudioAnalysisCache3\49aabdc27e233c764d5ffb81f6808f970.wfd, actual size 256579, dbsize 0, blocksetid: 4049651
... and 65 more. Run repair to fix it.
A repair --dry-run against a copied DB then refuses with:
ErrorID: DatabaseIsInRepairState
The database was attempted repaired, but the repair did not complete. This database may be incomplete and the repair process is not allowed to alter remote files as that could result in data loss.
Versions Tested During Recovery
The original Windows failure happened on:
2.3.0.0 (2.3.0.0_stable_2026-04-14), but the error tried to loadDuplicati.Library.Interface, Version=2.3.0.1.
On the Pi/Docker recovery side, I have tested:
-
duplicati/duplicati:beta- recreate completed but reported
DatabaseIsBrokenConsiderPurge purge-broken-files --dry-runfailed final validation withDatabaseInconsistencyrepair --dry-runfailed withDatabaseIsInRepairState
- recreate completed but reported
-
duplicati/duplicati:stable- tested against copied DBs
purge-broken-files --dry-runprocessed all36/36broken filesets- final validation still failed with
DatabaseInconsistency repair --dry-runalso failed withDatabaseIsInRepairState
I have not yet tested the latest canary on this recreated DB.
What I Need Help Deciding
At this point, I’m trying to choose the safest next recovery step. The current recreated database is in this state:
- Recreate completed, but reported 36 broken filelists.
purge-broken-files --dry-runsimulated all 36 purges.- Final validation still failed with
DatabaseInconsistency. repair --dry-runrefuses withDatabaseIsInRepairState.
My main questions are:
- Given that state, is the recommended next step to run a real
purge-broken-files, or does the dry-run validation failure mean that a real purge is unlikely to help? - If a real purge is not recommended, what is the next supported recovery step? I’m more than willing to lose 70 files if that makes the backup set usable again - but even the fact that I want to purge them doesn’t bypass the current blocker.
- Does
CUSTOMSQLITEOPTIONS_DUPLICATI=temp_store=FILEremain a reasonable workaround for large repair/recreate operations, or is there a newer recommended setting/version for this?
I have the backend data, recreated DB copies, and logs ready. I’m happy to provide specific command outputs or database logs if it helps track down where the logic is failing.
Thanks in advance for any help salvaging this. The frustrating part is that the apparent answer is acceptable to me - purge the broken files/filelists - but the current validation/repair-state loop is preventing me from getting to a usable backup set.