Hi ts678,
Thanks for the detailed feedback. I ran another generic rebuild from the web GUI, and here’s an excerpt from the verbose log:
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-if2cd360cc4c14027abf56e08b25f3f77.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-if2cd360cc4c14027abf56e08b25f3f77.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Processing indexlist volume 12874 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-i6aba8a89e87d4aeba68b05edfe16f799.dindex.zip.aes (18.278 KB)
5 Apr 2025 11:05 AM: Failed to process index file: duplicati-if021d78862ef46e5a42add193839d3e1.dindex.zip.aes
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-i6aba8a89e87d4aeba68b05edfe16f799.dindex.zip.aes (18.278 KB)
5 Apr 2025 11:05 AM: Processing indexlist volume 12873 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-if021d78862ef46e5a42add193839d3e1.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Failed to process index file: duplicati-iebb13c167017434db389193e7fabdb71.dindex.zip.aes
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-if021d78862ef46e5a42add193839d3e1.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Processing indexlist volume 12872 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-iebb13c167017434db389193e7fabdb71.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-iebb13c167017434db389193e7fabdb71.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Processing indexlist volume 12871 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-ic39bb3c5d4b64c3db5508b11be373f78.dindex.zip.aes (17.966 KB)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-ic39bb3c5d4b64c3db5508b11be373f78.dindex.zip.aes (17.966 KB)
5 Apr 2025 11:05 AM: Processing indexlist volume 12870 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-ieff656abe426471aaf0a601484685d25.dindex.zip.aes (37.653 KB)
5 Apr 2025 11:05 AM: Failed to process index file: duplicati-i8f73921cf2d34f78b5a7721cce4664a5.dindex.zip.aes
Duplicati.Library.Main.Volumes.InvalidManifestException: Invalid manifest detected, the field Blocksize has value 1048576 but the value 102400 was expected
at Duplicati.Library.Main.Volumes.VolumeBase.ManifestData.VerifyManifest(String manifest, Int64 blocksize, String blockhash, String filehash)
at Duplicati.Library.Main.Volumes.VolumeReaderBase.ReadManifests(Options options)
at Duplicati.Library.Main.Volumes.IndexVolumeReader..ctor(String compressor, String file, Options options, Int64 hashsize)
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-ieff656abe426471aaf0a601484685d25.dindex.zip.aes (37.653 KB)
5 Apr 2025 11:05 AM: Processing indexlist volume 12869 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-i8f73921cf2d34f78b5a7721cce4664a5.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-i8f73921cf2d34f78b5a7721cce4664a5.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Processing indexlist volume 12868 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-i4426dc44c89740e18a58cedcbc2ffd55.dindex.zip.aes (18.278 KB)
5 Apr 2025 11:05 AM: Failed to process index file: duplicati-i66d18145c80546058bda28d760867a02.dindex.zip.aes
Duplicati.Library.Main.Volumes.InvalidManifestException: Invalid manifest detected, the field Blocksize has value 1048576 but the value 102400 was expected
at Duplicati.Library.Main.Volumes.VolumeBase.ManifestData.VerifyManifest(String manifest, Int64 blocksize, String blockhash, String filehash)
at Duplicati.Library.Main.Volumes.VolumeReaderBase.ReadManifests(Options options)
at Duplicati.Library.Main.Volumes.IndexVolumeReader..ctor(String compressor, String file, Options options, Int64 hashsize)
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-i4426dc44c89740e18a58cedcbc2ffd55.dindex.zip.aes (18.278 KB)
5 Apr 2025 11:05 AM: Processing indexlist volume 12867 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-i66d18145c80546058bda28d760867a02.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Failed to process index file: duplicati-i2741af7a45ae4a1c94a65aab21ba20bc.dindex.zip.aes
Duplicati.Library.Main.Volumes.InvalidManifestException: Invalid manifest detected, the field Blocksize has value 1048576 but the value 102400 was expected
at Duplicati.Library.Main.Volumes.VolumeBase.ManifestData.VerifyManifest(String manifest, Int64 blocksize, String blockhash, String filehash)
at Duplicati.Library.Main.Volumes.VolumeReaderBase.ReadManifests(Options options)
at Duplicati.Library.Main.Volumes.IndexVolumeReader..ctor(String compressor, String file, Options options, Int64 hashsize)
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-i66d18145c80546058bda28d760867a02.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Processing indexlist volume 12866 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-i2741af7a45ae4a1c94a65aab21ba20bc.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-i2741af7a45ae4a1c94a65aab21ba20bc.dindex.zip.aes (845 bytes)
5 Apr 2025 11:05 AM: Processing indexlist volume 12865 of 42987
5 Apr 2025 11:05 AM: Backend event: Get - Completed: duplicati-i0a67f259c5c6488e8367159899e4b375.dindex.zip.aes (36.481 KB)
5 Apr 2025 11:05 AM: Failed to process index file: duplicati-ib3c54c7281f34c7a84de0c669713976a.dindex.zip.aes
Duplicati.Library.Main.Volumes.InvalidManifestException: Invalid manifest detected, the field Blocksize has value 1048576 but the value 102400 was expected
at Duplicati.Library.Main.Volumes.VolumeBase.ManifestData.VerifyManifest(String manifest, Int64 blocksize, String blockhash, String filehash)
at Duplicati.Library.Main.Volumes.VolumeReaderBase.ReadManifests(Options options)
at Duplicati.Library.Main.Volumes.IndexVolumeReader..ctor(String compressor, String file, Options options, Int64 hashsize)
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
5 Apr 2025 11:05 AM: Backend event: Get - Started: duplicati-i0a67f259c5c6488e8367159899e4b375.dindex.zip.aes (36.481 KB)
5 Apr 2025 11:05 AM: Processing indexlist volume 12864 of 42987
It looks like it’s skipping dindex files (which is expected) but then encounters several index files that fail with an InvalidManifestException
—specifically, the manifest’s Blocksize is 1048576
while 102400
was expected.
- The current database being rebuilt is based on 100 KiB blocks (
102400
bytes).
- But some
.dindex.zip.aes
files were created using 1 MiB blocks (1048576
bytes), which breaks consistency.
Do you have any insight into whether this blocksize discrepancy might be due to changes in the backup file format (perhaps a result of an update or downgrade)? I’m wondering if restoring from an older backup version (e.g. version “3”) might bypass this mismatch, as was suggested in the thread I saw earlier.
For additional context, my local machine (the one performing the rebuild) runs off an SSD. I attempted a restore on my Pi5—the second, stationary computer—but since it’s not a Windows machine (and I don’t have another Windows PC), that’s my only alternative to a portable laptop. The backups are stored on a home server—a Raspberry Pi 5 running Debian, with a 4TB USB3 hard drive connected to it—and that Pi is wired into my router via Ethernet. I also connect my laptop via Ethernet during rebuilds to maximise throughput. From the backup set, there are 21,549 dblock files, 42,987 dindex files, and 455 dlist files.
I also stand corrected in relation to the terminology. I’ve been attempting a rebuild (not a restore) and now understand that the restore tool may not be the one I need for regenerating the database.
About 3 weeks ago I attempted a restore directly on the Pi5 using the same exported configuration from my PC, adjusted for Linux paths. I did manage to rebuild the database in the Docker container, but the resulting SQLite file is only around 16 GB—compared to 30+ GB on my Windows machine. I’m unsure whether it restored everything correctly. I’ve since begun a repair there as well and will keep an eye on the verbose logs. I’ll also watch for any performance slowdown towards the latter stages of the rebuild, as you mentioned that the last 30% of the progress bar, which involves downloading dblock files, tends to slow dramatically.
I can accept that a full rebuild might take a week given the size of the backup; however, the inability to save progress makes Duplicati 2 feel fundamentally unsafe for large backups. Every interruption—whether it’s a reboot, a power cut, or a network issue—forces a complete restart from scratch. It’s not just inconvenient; it turns every rebuild into a high-stakes gamble, and without checkpointing or resume support, the odds aren’t in the user’s favour.
Regarding the age of the backup, I recognize that my most recent backup is now approaching two months old, which is risky. Creating a brand new backup might have to be the way to go, even though it would add roughly another terabyte to the 1.5 terabytes already used for the main backup.
Any further recommendations for workarounds to handle these manifest errors would be greatly appreciated. Thanks again for your help—I’m looking forward to any additional insights you might have!
Issue Raised |
Response/Plan |
Blocksize Discrepancy / InvalidManifestException |
Some index files use 1 MiB blocks (1048576 bytes) while the rebuild expects 100 KiB blocks (102400 bytes). Is this due to a format change (update/downgrade), or could restoring from an older version resolve it? |
Hardware/Environment |
I’m rebuilding on an SSD-equipped local machine and have also attempted a restore on my stationary Pi5 (non-Windows), as I don’t have another Windows PC. |
Terminology & Tool Usage |
I now understand I’m performing a rebuild (not a restore), and that the restore tool isn’t the correct option for regenerating the database. |
Performance Slowdown |
I’ll monitor for any slowdown in the final 30% of the rebuild (during dblock fetching) and report back any observations. |
Backup Age & New Backup Consideration |
With my latest backup nearing two months old, the risk is high. A new backup might be needed even if it adds roughly another terabyte to the storage. |
Safety & Checkpointing Concerns |
A full rebuild can take over a week, yet any interruption forces a complete restart—an absurd gamble for enterprise-level backups. Could the tech team offer any solutions to introduce checkpointing or resume support? |
Currently three days in to this round… I do need to use the laptop, but thankfully (and unusually) I don’t have to take it anywhere for work for another seven days. Fingers crossed the restore completes successfully and nothing goes wrong in the meantime.
Best,
Adam