ts678
March 21, 2025, 9:02pm
12
I’m keeping some of the historical evidence, but wanted to get the backup up again.
Set blocksize=100KB was the mystery ingredient (speculation above on why)
Recreate and get an error, so
purge-broken-files which runs nice and clean thanks to the blocksize fix.
Without that, it complains:
The operation PurgeBrokenFiles has failed with error: Found 1 file(s) with missing blocklist hashes => Found 1 file(s) with missing blocklist hashes
so I go to the database to (guessing) find my only file big enough to get two.
EDIT 1:
Found steps to reproduce the missing blocklist hashes error, and filed issue:
opened 12:41AM - 22 Mar 25 UTC
bug
- [x] I have searched open and closed issues for duplicates.
- [x] I have search… ed the [forum](https://forum.duplicati.com) for related topics.
----------------------------------------
## Environment info
- **Duplicati version**: 2.1.0.5 Stable
- **Operating system**: Windows 10
- **Backend**: N/A
## Description
2.1 purge-broken-files may fail with missing blocklist hashes if a large enough file is in a pre-2.1 backup. Probably one over 327,680,000 bytes will do it, as it will need a second blocklist at 2.0 default 100KB blocksize. 2.1 went to 1 MB default blocksize, where one blocklist can describe 34,359,738,368 bytes. Using an explicit `blocksize` option should avoid this issue. The problem was originally found with a backup that had a 331,350,016 byte file in it, and used default blocksize. While you can test with that, the described steps set up the failure using a smaller test file.
## Steps to reproduce
* Prepare a short text file and a file that contains over 32768 bytes
* Set blocksize=1KB for 2.0.8.1, making a blocklist per 32768 bytes
* Backup the larger file
* Add small file, backup
* Delete dindex and dblock of second backup
* Export job, import in 2.1.0.5
* Delete the blocksize override
* Recreate, and expect an error
Recreated database has missing blocks and 1 broken filelists. Consider using "list-broken-files" and "purge-broken-files" to purge broken data from the remote store and the database.
* purge-broken-files.
- **Actual result**:
```
Uploading file duplicati-20250321T234916Z.dlist.zip (873 bytes) ...
Deleting file duplicati-20250321T234915Z.dlist.zip ...
The operation PurgeBrokenFiles has failed with error: Found 1 file(s) with missing blocklist hashes => Found 1 file(s) with missing blocklist hashes
System.IO.InvalidDataException: Found 1 file(s) with missing blocklist hashes
at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
at Duplicati.Library.Main.Operation.PurgeBrokenFilesHandler.Run(IFilter filter)
at Duplicati.Library.Main.Controller.<>c__DisplayClass38_0.<PurgeBrokenFiles>b__0(PurgeBrokenFilesResults result)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
at Duplicati.Library.Main.Controller.RunAction[T](T result, Action`1 method)
at Duplicati.Library.Main.Controller.PurgeBrokenFiles(IFilter filter)
at Duplicati.CommandLine.Commands.PurgeBrokenFiles(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)
Return code: 100
```
- **Expected result**:
Just the purge, not the VerifyConsistency error
Repeating purge-broken-files at this point is refused
```
The operation PurgeBrokenFiles has failed with error: Cannot continue because the database is marked as being under repair, but does not have broken files. => Cannot continue because the database is marked as being under repair, but does not have broken files.
ErrorID: CannotListOnDatabaseInRepair
Cannot continue because the database is marked as being under repair, but does not have broken files.
Return code: 100
```
I didn't test a third purge-broken-files, but it looks like the one with the error fixed things so Recreate works now.
The behavior of the bug (as well as a look at the code) looks like purge-broken-files may be computing blockhash based on blocksize from an option value that's maybe set to the default if no user-supplied blocksize is available. What it should probably be using is the actual blocksize of the backup, e.g. as seen in job DB Configuration table.
I'll reference this issue from a forum topic which ran into this a lot. There I gave similar speculation to what's here.
## Screenshots
## Debug log