The errors you are seeing could be a symptom of this bug I fixed in a PR, but it has not been merged yet:
duplicati:master ← Jojo-1000:fix-repair-index-consistency
opened 08:40PM - 12 Dec 23 UTC
Closes #3202
[Forum discussion](https://forum.duplicati.com/t/any-way-to-rec… over-backup-repair-and-purge-broken-files-dont-help/17048/35?u=jojo-1000)
- Uses `GROUP BY Hash` to select any one of the possible blocklists in the database for each blocklist hash, so that re-use of blocks is ignored
- Add a test case to reproduce the bug
- Add a hash check when reading blocklists from index files, to prevent issues in recreate due to existing incorrect index files that might be caused by compact
## Steps to test recreate hash check
- Create unencrypted backup with at least one file larger than a block
- Make backup version
- Edit an index zip file on the destination, `list/[hash]` and duplicate the contents of the file (append to the back)
- Run database recreate
### Old behavior
- Recreate fails with error `Recreated database has missing blocks and 2 broken filelists. Consider using "list-broken-files" and "purge-broken-files" to purge broken data from the remote store and the database.`
- The database cannot be used to continue the backup
### New behavior
- Recreate finishes with warning `[file] had invalid blocklists which could not be used. Consider deleting this index file and run repair to recreate it.`
- The dblock file is used to get the correct blocklist
- The database can be used as normal
## Impact
This modifies a core SQL statement that will be used in repair, compact and to recreate missing index files. It should be tested that these work correctly for a backup with some data and a compact history.
The part from the bug discussion that might apply in your case:
I am quite surprised that this corruption bug is still not merged, but I haven’t followed the development closely for a while. @kenkendk maybe you could look at the PR for the next canary? It only lacks some testing with bigger databases, the root cause is well understood. Most of the changes are to detect corrupted blocklists in retrospect to fix existing corruption.
1 Like