Sorry that I didn’t provide more detailed information earlier.
It just took a better look at this problem.
Luckily, I just did read the other thread about very slow disaster recovery where people were complaining about the classic problem of files missing from restore and having issues with corrupted data blocks.
It reminded me about the broken logic of the program and after manually verifying, I found out that’s exactly the case.
I do get this error during restore:
Failed to patch with remote file: “duplicati-b43c50965b03842b0b63b7d11db7fea73.dblock.7z.aes”, message: System.IO.FileNotFoundException:
And when I run affected command with the filename:
A total of 310 file(s) are affected:
… And for sure … Those files which were missing from restore are in that specific archive …
I think this confirms that the problem isn’t related to paths / permissions at all.
There are a few main issues:
- Misleading or confusing error messages - If files can’t be restored due to dblock issues, why it’s not being clearly stated?
- Bad error detection logic - This error can be only detected when restoring
- Bad repair logic - Repair won’t do a thing about this error
- Bad recovery logic - The source files are STILL AVAILABLE, any time, when I run backup, the situation (at least for these specific files) could have been fixed by storing the dblocks again. But that won’t happen because the steps 2 and 3 fail.
Again this discussion wouldn’t even exist if those age old issues would have been fixed.
This is exactly the con side of de-duplication, especially when combined with many other things making it very brittle. Another big con of de-duplication is restore performance when the original database isn’t available.
I did get an error message. Which I pasted in this thread. But it seems that nobody else thought about that immediately. Because it was kind of obscure, and didn’t clearly indicate what the practical ramifications are. Also the information of known missing files wasn’t passed to the verification step.
I did once again run: test, list-broken-files, purge-broken-files, repair… And then backup… And even after these steps the restore of those files still fails, even if the files are still available from the source. Which just proves the point I made earlier. The situation would be (at least for these files) trivially fixed. Yet of course it still could and would affect files which aren’t available anymore.
Generic question about restore recovery
Based on this discussion and disaster recovery thoughts. What happens in situation where restoring a a large history log file or something like that fails due missing block(s). Does duplicati allow restoring partial files? What if from 10 GiB log file there’s a 100 KiB deduplication block which isn’t available. Does it prevent restoring the rest of the file or is there an option to allow restoring files which are known to be corrupted (or the missing parts filled with zero or so). Just wondering…
I know it’s generally not preferred ot restore known corrupt data. But in some situations, the actually very valuable data can very small part of a big data set. When that happens, allowing the restoration of known corrupted files would still be very helpful.