How to recover backup with damaged dblock file?

Sorry to jump in with no immediate help (those in this, please continue…) but I recently had the “Extra:” issue appear unexpectedly, so I took a technical look. Ultimately we want to understand the failure, so it can fixed…

duplicati-i03a0e9cd70ff44a292466283addb2b22.dindex.zip.aes: 20 errors
	Extra: +8vpCm4Muw2YV5qkqz6Phq37RmLDcioQjTVaEG0wtHQ=
	Extra: 0b8iZP2xvQ3bWfRsOkNVOGv+r9a7KChMFNlDfK0ixJw=
	Extra: 0mof4C7b71v6bcuQg8zDxWMeJ+Pq7JO21Knbzl7H824=
	Extra: 9CtMUlrfOMwd1vzyWXiK5yuwu6r5K2EN/YhXk2xy6e8=
	Extra: A91R/FY1WpWURm9TjA5SmPQBfn+LzG4/jBmrLbfqN4o=
	Extra: BfjzAmcEuBX7jYZJ+UoXeVpUgt+AScdWMx2Gn+1NoOc=
	Extra: FJzbL3YvV2CFDn0FT6ITrfaVI0Z4lNFHEArKgMj0h/w=
	Extra: QfeXtvB/L7IfvyKYvECeZj6BnEeBQ9hpTAXE2yIx9vo=
	Extra: bgG+tGJM2L3uysG01EBNAAuLYCgRKoZDWG+1wkW3sG4=
	Extra: e63i+inoVbTnXeaazsX15BLIXMJ+SMzvq2oCNNraIZs=
	... and 10 more


duplicati-b574cf4c8085e4c519d66c1e7c4974a66.dblock.zip.aes: 20 errors
	Extra: +8vpCm4Muw2YV5qkqz6Phq37RmLDcioQjTVaEG0wtHQ=
	Extra: 0b8iZP2xvQ3bWfRsOkNVOGv+r9a7KChMFNlDfK0ixJw=
	Extra: 0mof4C7b71v6bcuQg8zDxWMeJ+Pq7JO21Knbzl7H824=
	Extra: 9CtMUlrfOMwd1vzyWXiK5yuwu6r5K2EN/YhXk2xy6e8=
	Extra: A91R/FY1WpWURm9TjA5SmPQBfn+LzG4/jBmrLbfqN4o=
	Extra: BfjzAmcEuBX7jYZJ+UoXeVpUgt+AScdWMx2Gn+1NoOc=
	Extra: FJzbL3YvV2CFDn0FT6ITrfaVI0Z4lNFHEArKgMj0h/w=
	Extra: QfeXtvB/L7IfvyKYvECeZj6BnEeBQ9hpTAXE2yIx9vo=
	Extra: bgG+tGJM2L3uysG01EBNAAuLYCgRKoZDWG+1wkW3sG4=
	Extra: e63i+inoVbTnXeaazsX15BLIXMJ+SMzvq2oCNNraIZs=
	... and 10 more

What happened to me was that a compact put these blocks in a new dblock file, deleted some empty dblock files, then put these blocks into the next dblock file too. The database records them as belonging in the new file, so possibly I’m in the situation of it-works-but-it-complains, however I haven’t tried touching anything yet.

I downloaded dblock files, unencrypted, unzipped, and compared with WinMerge which identified all 20 dups. Unfortunately the deleted empty old dblock file wasn’t available for a certain look at what that had contained. Because the new dblock can only hold so much, at some time old dblock files can’t be fully emptied, so it will have to wait for the next dblock. I wasn’t sure how best to test a theory that the extras were a sequential run, however I tried sorting by sequential-looking ID in the Block table to see if extras were sequential. No, but in doing the lookup I saw they also weren’t Base64 of the SHA-256 hash I sometimes handle with tools like this. Instead, they were a character-mapped form, with a likely-troublesome forward slash becoming underscore, and plus becoming minus. The other characters in the Base64 table are letters and numbers and are safer.

What I find interesting is that all of my problems had this mapping, and those from the original post do too… Sampling my current block hashes, 49% have / and 49% have +, so it’s not rare, but to have 30 all show up this way, and 0 without the character mapping seems unlikely to be chance. Anyway, here are my 20 extras:

ID     Hash
355784 maGFet8d2IgpJQHiITVnsq-Ut-rA985ftbOl0lNfLao=
DB has maGFet8d2IgpJQHiITVnsq+Ut+rA985ftbOl0lNfLao=

355785 xVctf9vOEHW89Bk3xOzfiLADtczNMubXy-2LEF0AkDY=
DB has xVctf9vOEHW89Bk3xOzfiLADtczNMubXy+2LEF0AkDY=

355788 9CtMUlrfOMwd1vzyWXiK5yuwu6r5K2EN_YhXk2xy6e8=
DB has 9CtMUlrfOMwd1vzyWXiK5yuwu6r5K2EN/YhXk2xy6e8=

355789 e63i-inoVbTnXeaazsX15BLIXMJ-SMzvq2oCNNraIZs=
DB has e63i+inoVbTnXeaazsX15BLIXMJ+SMzvq2oCNNraIZs=

355790 vg3NXOItS6baRcHUCVPQrRyb69zKLXsbUvQLgv_ViBA=
DB has vg3NXOItS6baRcHUCVPQrRyb69zKLXsbUvQLgv/ViBA=

355791 FJzbL3YvV2CFDn0FT6ITrfaVI0Z4lNFHEArKgMj0h_w=
DB has FJzbL3YvV2CFDn0FT6ITrfaVI0Z4lNFHEArKgMj0h/w=

355792 goT-apmyALynpED2C2n0vUQ5Ua6TKk8cFuU5iJ5Ph28=
DB has goT+apmyALynpED2C2n0vUQ5Ua6TKk8cFuU5iJ5Ph28=

355793 0mof4C7b71v6bcuQg8zDxWMeJ-Pq7JO21Knbzl7H824=
DB has 0mof4C7b71v6bcuQg8zDxWMeJ+Pq7JO21Knbzl7H824=

355796 -8vpCm4Muw2YV5qkqz6Phq37RmLDcioQjTVaEG0wtHQ=
DB has +8vpCm4Muw2YV5qkqz6Phq37RmLDcioQjTVaEG0wtHQ=

355797 x2n8TLucw2rPJxEHwaBKS2SVS13z-hFmf4fM6oFGTUk=
DB has x2n8TLucw2rPJxEHwaBKS2SVS13z+hFmf4fM6oFGTUk=

355802 sGxIQhFRvtfH9ZaHv-tocxtXRyZe17tEiZVhRrPou3g=
DB has sGxIQhFRvtfH9ZaHv+tocxtXRyZe17tEiZVhRrPou3g=

355803 0b8iZP2xvQ3bWfRsOkNVOGv-r9a7KChMFNlDfK0ixJw=
DB has 0b8iZP2xvQ3bWfRsOkNVOGv+r9a7KChMFNlDfK0ixJw=

355804 BfjzAmcEuBX7jYZJ-UoXeVpUgt-AScdWMx2Gn-1NoOc=
DB has BfjzAmcEuBX7jYZJ+UoXeVpUgt+AScdWMx2Gn+1NoOc=

355805 bgG-tGJM2L3uysG01EBNAAuLYCgRKoZDWG-1wkW3sG4=
DB has bgG+tGJM2L3uysG01EBNAAuLYCgRKoZDWG+1wkW3sG4=

355817 A91R_FY1WpWURm9TjA5SmPQBfn-LzG4_jBmrLbfqN4o=
DB has A91R/FY1WpWURm9TjA5SmPQBfn+LzG4/jBmrLbfqN4o=

355818 f31-zQkFF37LIzGqG0H7aN9aj7M5h4UiGl0_OI-eDhs=
DB has f31+zQkFF37LIzGqG0H7aN9aj7M5h4UiGl0/OI+eDhs=

355819 QfeXtvB_L7IfvyKYvECeZj6BnEeBQ9hpTAXE2yIx9vo=
DB has QfeXtvB/L7IfvyKYvECeZj6BnEeBQ9hpTAXE2yIx9vo=

355820 myL8bELIN48gNgWmv31-vhjd8t4JyiXD2xXDX7FXFag=
DB has myL8bELIN48gNgWmv31+vhjd8t4JyiXD2xXDX7FXFag=

355821 i9eUzDmllQC9icHoFh5kU-OSOL2kvGrlQOJAJxQeuwc=
DB has i9eUzDmllQC9icHoFh5kU+OSOL2kvGrlQOJAJxQeuwc=

355823 ipvzQcfL4oolUhq-v_1TekFrFCpQzOGOU3ERZkkDUpE=
DB has ipvzQcfL4oolUhq+v/1TekFrFCpQzOGOU3ERZkkDUpE=

and if the guess about character mapping isn’t helpful, I have another guess to offer :wink:, which is that the dblock file that was disassembled for compaction contains an empty file (dealing a lot with those recently), which is apparently done wrong as 0 bytes packed size, according to 7-Zip whose test calls it a Data error. Windows File Explorer also complained about it, but I had it skip that file, and it kept on extracting the rest.

Here’s the log with an asterisk before the two mentioned files. This was another case of a hidden problem being inadvertently inserted into what seemed like a successful backup, until later work showed otherwise.

2018-12-17 20:32:47 -05 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there is 27.32% wasted space and the limit is 25%
2018-12-17 20:32:48 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-be7bc2b30f65a4cfa9500766aac617df1.dblock.zip.aes (16.27 MB)
2018-12-17 20:32:53 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-be7bc2b30f65a4cfa9500766aac617df1.dblock.zip.aes (16.27 MB)
2018-12-17 20:32:53 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b50f53e3980194661b927f067b4c96760.dblock.zip.aes (50.07 MB)
2018-12-17 20:33:07 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b50f53e3980194661b927f067b4c96760.dblock.zip.aes (50.07 MB)
2018-12-17 20:33:07 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b487235807edf44a1ab74ac1a8b102239.dblock.zip.aes (50.06 MB)
2018-12-17 20:33:19 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b487235807edf44a1ab74ac1a8b102239.dblock.zip.aes (50.06 MB)
2018-12-17 20:33:19 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b67b63d7968e349bfa7f24b3058c2a8f2.dblock.zip.aes (14.93 MB)
2018-12-17 20:33:22 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b67b63d7968e349bfa7f24b3058c2a8f2.dblock.zip.aes (14.93 MB)
2018-12-17 20:33:22 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bc84a3eeda0cc45f48678c5c97b574f4f.dblock.zip.aes (50.02 MB)
2018-12-17 20:33:38 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bc84a3eeda0cc45f48678c5c97b574f4f.dblock.zip.aes (50.02 MB)
2018-12-17 20:33:38 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b8a294b40162e4053998d1a3251809956.dblock.zip.aes (3.04 MB)
2018-12-17 20:33:40 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b8a294b40162e4053998d1a3251809956.dblock.zip.aes (3.04 MB)
2018-12-17 20:33:43 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b574cf4c8085e4c519d66c1e7c4974a66.dblock.zip.aes (50.04 MB)
*2018-12-17 20:35:25 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b574cf4c8085e4c519d66c1e7c4974a66.dblock.zip.aes (50.04 MB)
2018-12-17 20:35:25 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i03a0e9cd70ff44a292466283addb2b22.dindex.zip.aes (49.18 KB)
2018-12-17 20:35:26 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i03a0e9cd70ff44a292466283addb2b22.dindex.zip.aes (49.18 KB)
2018-12-17 20:35:27 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-be7bc2b30f65a4cfa9500766aac617df1.dblock.zip.aes (16.27 MB)
2018-12-17 20:35:27 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-be7bc2b30f65a4cfa9500766aac617df1.dblock.zip.aes (16.27 MB)
2018-12-17 20:35:27 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-ia3feabd6d45043b889fdaea2426c8312.dindex.zip.aes (13.89 KB)
2018-12-17 20:35:27 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-ia3feabd6d45043b889fdaea2426c8312.dindex.zip.aes (13.89 KB)
2018-12-17 20:35:27 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b50f53e3980194661b927f067b4c96760.dblock.zip.aes (50.07 MB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b50f53e3980194661b927f067b4c96760.dblock.zip.aes (50.07 MB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-ifdf5b6929b2749678ee50fa14450a34e.dindex.zip.aes (27.58 KB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-ifdf5b6929b2749678ee50fa14450a34e.dindex.zip.aes (27.58 KB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b487235807edf44a1ab74ac1a8b102239.dblock.zip.aes (50.06 MB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b487235807edf44a1ab74ac1a8b102239.dblock.zip.aes (50.06 MB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-icd2bbc0418044537a63989002fdc3a13.dindex.zip.aes (34.43 KB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-icd2bbc0418044537a63989002fdc3a13.dindex.zip.aes (34.43 KB)
2018-12-17 20:35:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b67b63d7968e349bfa7f24b3058c2a8f2.dblock.zip.aes (14.93 MB)
2018-12-17 20:35:29 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b67b63d7968e349bfa7f24b3058c2a8f2.dblock.zip.aes (14.93 MB)
2018-12-17 20:35:29 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-i852cb8aaf2c947c1a4d8abccafac3b85.dindex.zip.aes (8.84 KB)
2018-12-17 20:35:29 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-i852cb8aaf2c947c1a4d8abccafac3b85.dindex.zip.aes (8.84 KB)
2018-12-17 20:35:29 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b9296f0666088421e888e8ddc50db95a9.dblock.zip.aes (2.47 MB)
2018-12-17 20:35:32 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b9296f0666088421e888e8ddc50db95a9.dblock.zip.aes (2.47 MB)
2018-12-17 20:35:32 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-be09724cfc36f4301b332ac92f6391f4f.dblock.zip.aes (5.60 MB)
2018-12-17 20:35:34 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-be09724cfc36f4301b332ac92f6391f4f.dblock.zip.aes (5.60 MB)
2018-12-17 20:35:34 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b62f47a520ab34843b52d0a168a523dab.dblock.zip.aes (3.54 MB)
2018-12-17 20:35:36 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b62f47a520ab34843b52d0a168a523dab.dblock.zip.aes (3.54 MB)
2018-12-17 20:35:36 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b4cf18ef7c8464d1c89b761b2067386a9.dblock.zip.aes (3.16 MB)
2018-12-17 20:35:37 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b4cf18ef7c8464d1c89b761b2067386a9.dblock.zip.aes (3.16 MB)
2018-12-17 20:35:37 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-be4f42d062303483095dbf298737d00b5.dblock.zip.aes (4.29 MB)
2018-12-17 20:35:38 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-be4f42d062303483095dbf298737d00b5.dblock.zip.aes (4.29 MB)
2018-12-17 20:35:38 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-be82052fbf7384bef95c807ad171180f4.dblock.zip.aes (7.28 MB)
2018-12-17 20:35:40 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-be82052fbf7384bef95c807ad171180f4.dblock.zip.aes (7.28 MB)
*2018-12-17 20:35:42 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bf8ac8783fc394b22a22ca3eb888c68d9.dblock.zip.aes (34.35 MB)

Because this is sort of a production backup, I just exported it into a new job, so the old can be studied more.

EDIT: The theory about / and + translation is somewhat supported and somewhat refuted by other reports…

Question about TEST command vs. Backup with –full-remote-verification and backup-test-samples
10 extras, 0 without / or +

Verify a specific dblock.zip.aes file
8 extras, 4 without / or +

SharpCompress.Compressors.LZMA.DataErrorException
20 extras, 0 without / or +

“Unexpected number of remote volumes marked as deleted” error
8 extras, 2 without / or +, not counting a bizarre one with a dblock file name where the hash usually is.