Duplicati (MacOS) remote corruption repair failure

gctwnl · January 23, 2022, 9:21pm

A while back the remote storage for one of my backups had an issue and some data got corrupted. So, when I tried to run a backup, I now got:

The backup storage destination is missing data files. You can either enable –rebuild-missing-dblock-files or run the purge command to remove these files. The following files are missing: duplicati-b0a190a1cc8b1499283473f250edbd311.dblock.zip.aes, duplicati-ba3808cce103f42d083150420fb32aac2.dblock.zip.aes, duplicati-bc399283f5a29481e8344fe2a1a1bf975.dblock.zip.aes, duplicati-bb7cec6124b0147d5816396d90a9b2597.dblock.zip.aes, duplicati-b8007bd9a349f460fabbeedb602b63766.dblock.zip.aes, duplicati-be77e5c8861ae488bb25d887ea668dc95.dblock.zip.aes, duplicati-b654db1068d684162a3601677510beafc.dblock.zip.aes, duplicati-bc98344087c354250864acca0ed4128ce.dblock.zip.aes, duplicati-b711d5b58167144e9bdec6ed8433ca895.dblock.zip.aes, duplicati-bb39c353f26464970a8439fd8c94dc409.dblock.zip.aes, duplicati-bf481caa949f34f0689c9f6584073a9f0.dblock.zip.aes, duplicati-b6b81f98b2b6544cca2acc33e306ea60f.dblock.zip.aes

So, I tried to follow repair instructions but failed:

sudo mono /Applications/Duplicati.app/Contents/Resources/Duplicati.CommandLine.exe backup "s3s URL" /Users/lia/Documents/ /Users/lia/Pictures/ /Users/lia/Desktop/ --backup-name=ComputerLia --dbpath=/var/root/.config/Duplicati/CRQENUZKMY.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase=[snip] --retention-policy="1W:1D,4W:1W,12M:1M" --disable-module=console-password-input --rebuild-missing-dblock-files
Password:
Backup started at 1/23/2022 9:10:29 PM
Checking remote backup ...
  Listing remote folder ...
Missing file: duplicati-20201007T101330Z.dlist.zip.aes
Missing file: duplicati-20201111T144115Z.dlist.zip.aes
Missing file: duplicati-20201212T190109Z.dlist.zip.aes
Missing file: duplicati-b0a190a1cc8b1499283473f250edbd311.dblock.zip.aes
Missing file: duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes
Missing file: duplicati-20210825T193153Z.dlist.zip.aes
Missing file: duplicati-20210902T091337Z.dlist.zip.aes
Missing file: duplicati-ba3808cce103f42d083150420fb32aac2.dblock.zip.aes
Missing file: duplicati-i6dcaa73d27c64a5895abaa53592b0d5f.dindex.zip.aes
Missing file: duplicati-bc399283f5a29481e8344fe2a1a1bf975.dblock.zip.aes
Missing file: duplicati-i6c9c0305def64d37b7f5e3582996b244.dindex.zip.aes
Missing file: duplicati-20210908T145543Z.dlist.zip.aes
Missing file: duplicati-20210909T154257Z.dlist.zip.aes
Missing file: duplicati-20210910T183622Z.dlist.zip.aes
Missing file: duplicati-20210912T165709Z.dlist.zip.aes
Missing file: duplicati-20210914T094808Z.dlist.zip.aes
Missing file: duplicati-bb7cec6124b0147d5816396d90a9b2597.dblock.zip.aes
Missing file: duplicati-ie376151220764aa3ad8195224a1569f1.dindex.zip.aes
Missing file: duplicati-b8007bd9a349f460fabbeedb602b63766.dblock.zip.aes
Missing file: duplicati-id80cab85c36d4db691a126740047c3ea.dindex.zip.aes
Missing file: duplicati-be77e5c8861ae488bb25d887ea668dc95.dblock.zip.aes
Missing file: duplicati-ia4e875a42ccb4e2aaa77eefee44ea3e1.dindex.zip.aes
Missing file: duplicati-b654db1068d684162a3601677510beafc.dblock.zip.aes
Missing file: duplicati-i85ebdfd818174ee7a8d0c02e209f6180.dindex.zip.aes
Missing file: duplicati-bc98344087c354250864acca0ed4128ce.dblock.zip.aes
Missing file: duplicati-iefab6e0f6a0a4ea0b8f8c153cd96909d.dindex.zip.aes
Missing file: duplicati-20210915T102025Z.dlist.zip.aes
Missing file: duplicati-b711d5b58167144e9bdec6ed8433ca895.dblock.zip.aes
Missing file: duplicati-ia0dde18f3ec44505ae700fb5accd69b4.dindex.zip.aes
Missing file: duplicati-bb39c353f26464970a8439fd8c94dc409.dblock.zip.aes
Missing file: duplicati-i5581506ad0614082b9c64bb1c0204996.dindex.zip.aes
Missing file: duplicati-bf481caa949f34f0689c9f6584073a9f0.dblock.zip.aes
Missing file: duplicati-i155c1ec271b048e3b73004df3c8d5579.dindex.zip.aes
Missing file: duplicati-20210915T125147Z.dlist.zip.aes
Missing file: duplicati-b6b81f98b2b6544cca2acc33e306ea60f.dblock.zip.aes
Missing file: duplicati-i4b7dc6bdb84d4f40ad89ced0d8fc2baa.dindex.zip.aes
Found 36 files that are missing from the remote storage, please run repair
Fatal error => Found 36 files that are missing from the remote storage, please run repair
ErrorID: MissingRemoteFiles
Found 36 files that are missing from the remote storage, please run repair

Computer-van-Lia:~ bh$ sudo mono /Applications/Duplicati.app/Contents/Resources/Duplicati.CommandLine.exe --dry-run purge-broken-files "s3s URL" /Users/lia/Documents/ /Users/lia/Pictures/ /Users/lia/Desktop/ --backup-name=ComputerLia --dbpath=/var/root/.config/Duplicati/CRQENUZKMY.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase=[snip] --retention-policy="1W:1D,4W:1W,12M:1M" --disable-module=console-password-input 
Found 4 commands but expected 1, commands: 
"s3s URL"
"/Users/lia/Documents/"
"/Users/lia/Pictures/"
"/Users/lia/Desktop/"
Computer-van-Lia:~ bh$ sudo mono /Applications/Duplicati.app/Contents/Resources/Duplicati.CommandLine.exe --dry-run purge-broken-files --backup-name=ComputerLia --dbpath=/var/root/.config/Duplicati/CRQENUZKMY.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase=aJDWrywHXZcdtrUR7CZyd3xQ --retention-policy="1W:1D,4W:1W,12M:1M" --disable-module=console-password-input 
Found 0 commands but expected 1, commands: 

Computer-van-Lia:~ bh$ sudo mono /Applications/Duplicati.app/Contents/Resources/Duplicati.CommandLine.exe purge-broken-files "s3s URL" /Users/lia/Documents/ /Users/lia/Pictures/ /Users/lia/Desktop/ --dry-run --backup-name=ComputerLia --dbpath=/var/root/.config/Duplicati/CRQENUZKMY.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase=[snip] --retention-policy="1W:1D,4W:1W,12M:1M" --disable-module=console-password-input 
Found 4 commands but expected 1, commands: 
"s3s URL"
"/Users/lia/Documents/"
"/Users/lia/Pictures/"
"/Users/lia/Desktop/"
Computer-van-Lia:~ bh$ sudo mono /Applications/Duplicati.app/Contents/Resources/Duplicati.CommandLine.exe create-report "s3s URL" /Users/lia/Documents/ /Users/lia/Pictures/ /Users/lia/Desktop/ --backup-name=ComputerLia --dbpath=/var/root/.config/Duplicati/CRQENUZKMY.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase=[snip] --retention-policy="1W:1D,4W:1W,12M:1M" --disable-module=console-password-input 

System.NotSupportedException: Attempted to write a stream that is larger than 4GiB without setting the zip64 option
  at SharpCompress.Writers.Zip.ZipWriter+ZipWritingStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00034] in <5717dfb1db2745ffb30a27e1fee78b19>:0 
  at Duplicati.Library.Utility.Utility.CopyStream (System.IO.Stream source, System.IO.Stream target, System.Boolean tryRewindSource, System.Byte[] buf) [0x0002f] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.Utility.CopyStream (System.IO.Stream source, System.IO.Stream target) [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Main.Operation.CreateBugReportHandler.Run () [0x0018c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass27_0.<CreateLogDatabase>b__0 (Duplicati.Library.Main.CreateLogDatabaseResults result) [0x00019] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, System.Action`1[T] method) [0x00002] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.CreateLogDatabase (System.String targetpath) [0x00023] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.CommandLine.Commands.CreateBugReport (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x000b9] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00342] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0 
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0

Basically, even after looking at the manual, I am at a loss how to get this backup in a working condition again. Basically, I want to keep as much as possible of what is in the remote storage and get the backup back in working condition. How, though?

gctwnl · January 24, 2022, 9:19pm

Nobody? I really have no idea how to fix this situation and the suggestions Duplicati has thrown up haven’t worked.

drwtsn32 · January 25, 2022, 12:12am

You actually ran the backup command, not repair. Can you try again with the repair command? Note that for the repair command you should leave off some of the command line arguments, namely the ones that show what folders you want to back up. So remove these: /Users/lia/Documents/ /Users/lia/Pictures/ /Users/lia/Desktop/

gctwnl · January 25, 2022, 9:05pm

Thanks. I did repair. That did not work but I was able to do a --rebuild-missing-dblock-files for part of the repair and in the end had to run purge-missing-files. The back-up works again, except for a series of 2022-01-25 21:17:58 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes is listed as Verified with size 414157 but should be 65773, please verify the sha256 hash "pufY+BLcm5wBSPCWGhyaWLkd32RUYwG7DdAG33Kxg1M=", warnings

drwtsn32 · January 26, 2022, 12:19am

Can you elaborate on this… how did it not work? Did you get an error message?

The fact that you still get warnings bothers me. IMO if the back end data gets corrupted, I might consider starting over (after fixing whatever caused the back end corruption… hardware issue?)

gctwnl · January 26, 2022, 9:31am

I first did a basic repair, that failed (partly, maybe) and told be to try --rebuild-missing-dblock-files
I did that and that failed (again, partly maybe) and was told to do a purge-broken-files
(I did not save those messages, they were in a shell and I forgot to save the output)
I did that and the backup is now working again, but it has these warnings. A verify command shows them as well.

The origin of all of this was hardware failure behind the remote MinIO S3 backend that I am connecting to (which I also manage in full, hardware and up). There were initial signs of error on the remote hardware (warnings from DriveDx). As quickly as possible, the hardware was replaced and the remote data was moved onto new (RAID) storage. But some of it apparently did get damaged. Two of the backups were hit. One was reasonable easy to repair. This one isn’t.

Starting anew is something I am trying to prevent because I would like to keep old data in the backup. The message please verify the sha256 hash has me stumped as I don’t know what Duplicati wants me to do exactly. It seems maybe the block was recreated, but the Duplicati database still has outdated size information?

On the backend (this is the file from the example warning above):

% ls -l duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes
-rw-r--r--  1 root  wheel  414157 Jan 25 20:38 duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes

drwtsn32 · January 26, 2022, 3:58pm

I believe it is asking you to validate that the file is actually damaged. You could run sha256sum to validate that Duplicati is reporting the incorrect value properly. If it really is damaged (which it probably is), I think the correct course of action is to delete the file and then run the purge-broken-files command again. It should resolve the warning message, at the expense of losing some backup data.

The affected command can help you see what source files are affected by corrupt back end files.

Here’s some more information on how to recover when back end files get damaged or deleted:

ts678 · January 27, 2022, 12:08am

How big are your large dblock and dindex files? This seems awfully large for a dindex file.
The latter number might be about right for a 100 MB remote volume (dblock) size though.

I notice that this file was among the missing at one time. Doing repair can replace dindex and dlist files from database records (assuming database is intact). The dblock files are the hard ones, as they’re source data, with blocks frequently no longer in existence because the source file changed. Rebuilding tends not to work.

If your database seems intact (quite possible until you try to recreate it – save old DB in case recreate fails) then verify of all files might show which ones are no longer readable, due to either hash or decryption failure. Basically, a damage assessment might be in order. Hopefully it’s limited, but it’d be better to find out sooner.

gctwnl · January 27, 2022, 9:06pm

Is ‘running a sha256sum’ the same as using openssl sha256 on the backend file? Because I tried that, but the value seem completely of a different format.

gctwnl · January 27, 2022, 9:12pm

ts678:

gctwnl:

listed as Verified with size 414157 but should be 65773

How big are your large dblock and dindex files? This seems awfully large for a dindex file.
The latter number might be about right for a 100 MB remote volume (dblock) size though.

I notice that this file was among the missing at one time. Doing repair can replace dindex and dlist files from database records (assuming database is intact). The dblock files are the hard ones, as they’re source data, with blocks frequently no longer in existence because the source file changed. Rebuilding tends not to work.

gctwnl:

A verify command shows them as well.

If your database seems intact (quite possible until you try to recreate it – save old DB in case recreate fails) then verify of all files might show which ones are no longer readable, due to either hash or decryption failure. Basically, a damage assessment might be in order. Hopefully it’s limited, but it’d be better to find out sooner.

My blocks are 50MB (standard size)

I could not parse that last paragraph. You want me to save the old database and recreate it?

The local database has options repair, delete, and recreate (delete and repair). As well as (below the local database path) reset, save, save and repair, and move existing database, all the latter 4 are greyed out.

Should I use a shell to make a copy of the database, then attempt things?

ts678 · January 27, 2022, 9:19pm

No. Do a big verify to find out what’s broken.

gctwnl · January 27, 2022, 9:19pm

The files it is complaining about are simply dindex files. Can I not just simply remove these from the backend and get them recreated?

gctwnl · January 27, 2022, 9:25pm

Does a ‘big verify’ differ from a ‘verify’? Because I already did a verify --rebuild-missing-dblock-files --full-result:

Listing remote folder ...
remote file duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes is listed as Verified with size 414157 but should be 65773, please verify the sha256 hash "pufY+BLcm5wBSPCWGhyaWLkd32RUYwG7DdAG33Kxg1M="
remote file duplicati-i6dcaa73d27c64a5895abaa53592b0d5f.dindex.zip.aes is listed as Verified with size 465421 but should be 60733, please verify the sha256 hash "C/AbkCuzdaozhpqaKYc3gL+XL8XeM8tLA4wNUB07kCU="
remote file duplicati-i6c9c0305def64d37b7f5e3582996b244.dindex.zip.aes is listed as Verified with size 261533 but should be 34381, please verify the sha256 hash "AAcsyamWINGP5v6/wNDM0dneadiCTrw9BuTr7WvqbhQ="
remote file duplicati-ie376151220764aa3ad8195224a1569f1.dindex.zip.aes is listed as Verified with size 107709 but should be 14077, please verify the sha256 hash "GS6zoDR60GvESGfiMdPU0shcEwBilSHkNJecaWf0bGw="
remote file duplicati-id80cab85c36d4db691a126740047c3ea.dindex.zip.aes is listed as Verified with size 893 but should be 1917, please verify the sha256 hash "raBg09ee1bfJic5ywkSNlFmUz22OOuom/IqAJqITNLU="
remote file duplicati-ia4e875a42ccb4e2aaa77eefee44ea3e1.dindex.zip.aes is listed as Verified with size 7805 but should be 7917, please verify the sha256 hash "QRwaN2X7sk6poHta6Uhmh9TqqILMGJUSoaxQa6EEXb8="
remote file duplicati-iefab6e0f6a0a4ea0b8f8c153cd96909d.dindex.zip.aes is listed as Verified with size 1037 but should be 1053, please verify the sha256 hash "Az8CSnQKXSz/uqVryghDMWL0rsAUSooaulMLLSZrA2g="
remote file duplicati-i5581506ad0614082b9c64bb1c0204996.dindex.zip.aes is listed as Verified with size 3245 but should be 3357, please verify the sha256 hash "FIB6NhsGdwBCX8DSwnauRDzVpWnhRJk/9hrz2vttFgc="
Downloading file (18.00 MB) ...
Downloading file (17.97 KB) ...
Downloading file (49.92 MB) ...
Examined 3 files and found no errors

ts678 · January 27, 2022, 9:28pm

Assuming your database is intact, this should work. You can save old one elsewhere if you wish, for later inspection if needed. I don’t know what would make it that large. Your database is probably intact, unless recreate from a known-broken destination has been tried, in which case there’s no telling if it even works.

3 files is not all your files. If you have more, test them to see whether they suffered damage in the failure.

The TEST command

<samples> specifies the number of samples to be tested. If “all” is specified, all files in the backup will be tested.

gctwnl · January 27, 2022, 10:00pm

Running (test all). It’s going to take a while. Do you want the entire output here or just the failures? The large dindex files seem to be OK.

ts678 · January 27, 2022, 10:05pm

Just the failures or maybe less, maybe even summarized. This is to estimate extent and chance of a fixup.

gctwnl · January 28, 2022, 11:34am

Result of a test all --full-result (errors only):

Listing remote folder ...
remote file duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes is listed as Verified with size 414157 but should be 65773, please verify the sha256 hash "pufY+BLcm5wBSPCWGhyaWLkd32RUYwG7DdAG33Kxg1M="
remote file duplicati-i6dcaa73d27c64a5895abaa53592b0d5f.dindex.zip.aes is listed as Verified with size 465421 but should be 60733, please verify the sha256 hash "C/AbkCuzdaozhpqaKYc3gL+XL8XeM8tLA4wNUB07kCU="
remote file duplicati-i6c9c0305def64d37b7f5e3582996b244.dindex.zip.aes is listed as Verified with size 261533 but should be 34381, please verify the sha256 hash "AAcsyamWINGP5v6/wNDM0dneadiCTrw9BuTr7WvqbhQ="
remote file duplicati-ie376151220764aa3ad8195224a1569f1.dindex.zip.aes is listed as Verified with size 107709 but should be 14077, please verify the sha256 hash "GS6zoDR60GvESGfiMdPU0shcEwBilSHkNJecaWf0bGw="
remote file duplicati-id80cab85c36d4db691a126740047c3ea.dindex.zip.aes is listed as Verified with size 893 but should be 1917, please verify the sha256 hash "raBg09ee1bfJic5ywkSNlFmUz22OOuom/IqAJqITNLU="
remote file duplicati-ia4e875a42ccb4e2aaa77eefee44ea3e1.dindex.zip.aes is listed as Verified with size 7805 but should be 7917, please verify the sha256 hash "QRwaN2X7sk6poHta6Uhmh9TqqILMGJUSoaxQa6EEXb8="
remote file duplicati-iefab6e0f6a0a4ea0b8f8c153cd96909d.dindex.zip.aes is listed as Verified with size 1037 but should be 1053, please verify the sha256 hash "Az8CSnQKXSz/uqVryghDMWL0rsAUSooaulMLLSZrA2g="
remote file duplicati-i5581506ad0614082b9c64bb1c0204996.dindex.zip.aes is listed as Verified with size 3245 but should be 3357, please verify the sha256 hash "FIB6NhsGdwBCX8DSwnauRDzVpWnhRJk/9hrz2vttFgc="

...

Failed to process file duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes => The file duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes was downloaded and had size 414157 but the size was expected to be 65773
Failed to process file duplicati-i6dcaa73d27c64a5895abaa53592b0d5f.dindex.zip.aes => The file duplicati-i6dcaa73d27c64a5895abaa53592b0d5f.dindex.zip.aes was downloaded and had size 465421 but the size was expected to be 60733
Failed to process file duplicati-i6c9c0305def64d37b7f5e3582996b244.dindex.zip.aes => The file duplicati-i6c9c0305def64d37b7f5e3582996b244.dindex.zip.aes was downloaded and had size 261533 but the size was expected to be 34381
Failed to process file duplicati-ie376151220764aa3ad8195224a1569f1.dindex.zip.aes => The file duplicati-ie376151220764aa3ad8195224a1569f1.dindex.zip.aes was downloaded and had size 107709 but the size was expected to be 14077
Failed to process file duplicati-ia4e875a42ccb4e2aaa77eefee44ea3e1.dindex.zip.aes => The file duplicati-ia4e875a42ccb4e2aaa77eefee44ea3e1.dindex.zip.aes was downloaded and had size 7805 but the size was expected to be 7917
Failed to process file duplicati-id80cab85c36d4db691a126740047c3ea.dindex.zip.aes => The file duplicati-id80cab85c36d4db691a126740047c3ea.dindex.zip.aes was downloaded and had size 893 but the size was expected to be 1917
Failed to process file duplicati-i85ebdfd818174ee7a8d0c02e209f6180.dindex.zip.aes => Hash mismatch on file "/tmp/dup-aca33395-664e-42a9-aeab-d388cb0ec910", recorded hash: UIx96GeBDItG2SyMBlck7vtFt7WI3jkweaqnXxvQ/9A=, actual hash mzVbaaIiCaKKqneXAgTyUVYgCC4ZIIsU8uRxwrVgeZM=
Failed to process file duplicati-i4b7dc6bdb84d4f40ad89ced0d8fc2baa.dindex.zip.aes => Hash mismatch on file "/tmp/dup-47b94edd-8734-42c4-a00d-ad558976e2f4", recorded hash: FTPQbzQCWfAFyGP1e8i8rvBudISpSBqyVdb/8uswJII=, actual hash pp22iHYu4s2cJNWZGx5MM4hInIt5cGBqpp+fVXVedDE=
Failed to process file duplicati-i155c1ec271b048e3b73004df3c8d5579.dindex.zip.aes => Hash mismatch on file "/tmp/dup-f5d9f008-7dd6-499f-8198-7b4925153caf", recorded hash: Ub4UMdyMBBE3Lugkwvs0oTGxg/J9aHufVy13/kKBDRQ=, actual hash 4xq/fv2ru5V+O88mXALAWuyT7fxmjb8V5wTTEPj+kAU=
Failed to process file duplicati-i5581506ad0614082b9c64bb1c0204996.dindex.zip.aes => The file duplicati-i5581506ad0614082b9c64bb1c0204996.dindex.zip.aes was downloaded and had size 3245 but the size was expected to be 3357
Failed to process file duplicati-iefab6e0f6a0a4ea0b8f8c153cd96909d.dindex.zip.aes => The file duplicati-iefab6e0f6a0a4ea0b8f8c153cd96909d.dindex.zip.aes was downloaded and had size 1037 but the size was expected to be 1053
Failed to process file duplicati-ia0dde18f3ec44505ae700fb5accd69b4.dindex.zip.aes => Hash mismatch on file "/tmp/dup-50e79429-529a-4bb3-8625-77309258664f", recorded hash: Kz3bmFsJJjnjWAYMhfh1w0/Hhp33e1J156YQIwNxuIs=, actual hash 79bPeHiahdBw0DPUqpmWW3ZfFQ1Fs8Q4OdiD2LPnq6c=

...

duplicati-i2d6748dfee094b21993ad616dccaaf28.dindex.zip.aes: 4 errors
	Extra: /PDkQd1gZmrT3jchzCp9dCL8ioT5P81pSI7Z9cNbUiA=
	Extra: j/q2bKamD1/YEbpoxxOlP5mZfiafLXqnb9Qb23kmMP4=
	Extra: jJm/eu32lXjndh5sIlcbF2/VOoF7scuq0K3EQzV8IVM=
	Extra: xorXOYRr067CTHjbRpDswwN8WPkLBADdOAGW7SYrRAM=
duplicati-ie06cc053f19341f690913c6d95a0c17d.dindex.zip.aes: 4 errors
	Extra: +uSZCDwpIFHvEtkc2P02CmMu15NwgYXuSI44lrCfMIk=
	Extra: QI6of+BCoBkOvwcpbbcbDEO+x3GoocnhyKeShWC4rgk=
	Extra: iFP7fD4LLdITl/6rFYQsbFn7yX0+hcynEDCe6+YktBU=
	Extra: vbdp1uUP6VxgpaeG+cbDNmP/pNbvpC50O023A2NTj+Y=
duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes: 1 errors
	Error: The file duplicati-i6fb1c4f8179a4ded932066deab5b58d9.dindex.zip.aes was downloaded and had size 414157 but the size was expected to be 65773
duplicati-i6dcaa73d27c64a5895abaa53592b0d5f.dindex.zip.aes: 1 errors
	Error: The file duplicati-i6dcaa73d27c64a5895abaa53592b0d5f.dindex.zip.aes was downloaded and had size 465421 but the size was expected to be 60733
duplicati-i6c9c0305def64d37b7f5e3582996b244.dindex.zip.aes: 1 errors
	Error: The file duplicati-i6c9c0305def64d37b7f5e3582996b244.dindex.zip.aes was downloaded and had size 261533 but the size was expected to be 34381
duplicati-ie376151220764aa3ad8195224a1569f1.dindex.zip.aes: 1 errors
	Error: The file duplicati-ie376151220764aa3ad8195224a1569f1.dindex.zip.aes was downloaded and had size 107709 but the size was expected to be 14077
duplicati-ia4e875a42ccb4e2aaa77eefee44ea3e1.dindex.zip.aes: 1 errors
	Error: The file duplicati-ia4e875a42ccb4e2aaa77eefee44ea3e1.dindex.zip.aes was downloaded and had size 7805 but the size was expected to be 7917
duplicati-id80cab85c36d4db691a126740047c3ea.dindex.zip.aes: 1 errors
	Error: The file duplicati-id80cab85c36d4db691a126740047c3ea.dindex.zip.aes was downloaded and had size 893 but the size was expected to be 1917
duplicati-i85ebdfd818174ee7a8d0c02e209f6180.dindex.zip.aes: 1 errors
	Error: Hash mismatch on file "/tmp/dup-aca33395-664e-42a9-aeab-d388cb0ec910", recorded hash: UIx96GeBDItG2SyMBlck7vtFt7WI3jkweaqnXxvQ/9A=, actual hash mzVbaaIiCaKKqneXAgTyUVYgCC4ZIIsU8uRxwrVgeZM=
duplicati-i4b7dc6bdb84d4f40ad89ced0d8fc2baa.dindex.zip.aes: 1 errors
	Error: Hash mismatch on file "/tmp/dup-47b94edd-8734-42c4-a00d-ad558976e2f4", recorded hash: FTPQbzQCWfAFyGP1e8i8rvBudISpSBqyVdb/8uswJII=, actual hash pp22iHYu4s2cJNWZGx5MM4hInIt5cGBqpp+fVXVedDE=
duplicati-i155c1ec271b048e3b73004df3c8d5579.dindex.zip.aes: 1 errors
	Error: Hash mismatch on file "/tmp/dup-f5d9f008-7dd6-499f-8198-7b4925153caf", recorded hash: Ub4UMdyMBBE3Lugkwvs0oTGxg/J9aHufVy13/kKBDRQ=, actual hash 4xq/fv2ru5V+O88mXALAWuyT7fxmjb8V5wTTEPj+kAU=
duplicati-i5581506ad0614082b9c64bb1c0204996.dindex.zip.aes: 1 errors
	Error: The file duplicati-i5581506ad0614082b9c64bb1c0204996.dindex.zip.aes was downloaded and had size 3245 but the size was expected to be 3357
duplicati-iefab6e0f6a0a4ea0b8f8c153cd96909d.dindex.zip.aes: 1 errors
	Error: The file duplicati-iefab6e0f6a0a4ea0b8f8c153cd96909d.dindex.zip.aes was downloaded and had size 1037 but the size was expected to be 1053
duplicati-ia0dde18f3ec44505ae700fb5accd69b4.dindex.zip.aes: 1 errors
	Error: Hash mismatch on file "/tmp/dup-50e79429-529a-4bb3-8625-77309258664f", recorded hash: Kz3bmFsJJjnjWAYMhfh1w0/Hhp33e1J156YQIwNxuIs=, actual hash 79bPeHiahdBw0DPUqpmWW3ZfFQ1Fs8Q4OdiD2LPnq6c=
duplicati-bdfc2ca2198d14380b1a7ced3d84759a6.dblock.zip.aes: 4 errors
	Extra: +uSZCDwpIFHvEtkc2P02CmMu15NwgYXuSI44lrCfMIk=
	Extra: QI6of+BCoBkOvwcpbbcbDEO+x3GoocnhyKeShWC4rgk=
	Extra: iFP7fD4LLdITl/6rFYQsbFn7yX0+hcynEDCe6+YktBU=
	Extra: vbdp1uUP6VxgpaeG+cbDNmP/pNbvpC50O023A2NTj+Y=
duplicati-b96c36c01520848dead29346e4efbe36d.dblock.zip.aes: 4 errors
	Extra: /PDkQd1gZmrT3jchzCp9dCL8ioT5P81pSI7Z9cNbUiA=
	Extra: j/q2bKamD1/YEbpoxxOlP5mZfiafLXqnb9Qb23kmMP4=
	Extra: jJm/eu32lXjndh5sIlcbF2/VOoF7scuq0K3EQzV8IVM=
	Extra: xorXOYRr067CTHjbRpDswwN8WPkLBADdOAGW7SYrRAM=

So, it seems two dblock files are problematic and quite q few more dindex files. On a 200GB size, this is 100MB actual data, or 2 out of 2422 dblocks.

ts678 · January 28, 2022, 2:56pm

The problem might be rather mild or maybe harmless. At least the file is there and can be decrypted. The “Extra” errors historically have seemed benign (probably better than “Missing” errors) and tend to come in pairs, with a dindex and its associated dblock both complaining. At first glance, your Extras fit that pattern.

Although I haven’t been able to follow the code, a look at one of mine suggests that “Extra” in dindex says dindex internal index is claiming a block (known by hash) that doesn’t belong in the associated dblock file. “Extra” in dblock says that a block is in the dblock file that records say shouldn’t be there. In my case, the block was stored and available, per DB records, but there was also an extra copy of the block elsewhere.

Your best plan might be to make a backup copy of the database, delete (or move to another folder, just in case) bad dindex files, let Repair make them again, see what came out different on them, see if all is OK.

gctwnl · January 29, 2022, 5:59pm

Very funny:

Stopped the duplicati service
Made a backup copy of the slit database for the affected backup
Moved the affected dindex files (copied them to another location, told MinIO to remove them)
Started the duplicati service, planning to run a repair command
Logged in to the web interface
But before i got to that, duplicati had already started to attempt to backup
Which succeeded without warnings…

Any next backup just works. No warning about missing dindex. I just started another test all.

gctwnl · January 29, 2022, 11:26pm

Two dindex files with the ‘extras’ had escaped the purge. I removed these, ran a repair (which uploaded two files) and am now running a test all again