Will Duplicati do anything with the knowledge that these are these corrupted blocks, if they are used to represent current data? E.g. attempt to upload a block to replace it? Or is the backup hosed and a new set needs to be created?
During a restore any corrupted blocks are written to disk as zeros - so as much of a file as possible will be restored, but usability of such as file varies by program and where the corrupt parts are.
The backup isn’t hosed - just those specific blocks. If the source file blocks stored in those corrupted locations are changed, the new versions of the blocks will be backed up - but the old hosed versions will also still be there. (At least until your “Keep until” rules are hit.)
If the source files aren’t changing or you just feel better not having corrupt blocks in your backup, you can tell duplicati to purge those files from the backup in which case on the next run it should start backing them up again, but without any of the purged history. (I think you can also delete specific versions of files if you want to do it that way…)
2 Questions:
Oh, glad to see these tools exist. Two questiosn:
-
Is there a difference between --purge-broken-files and deleting the offending blocks and running repair?
-
Has the issue of not recreating previous back up sets, given those files are unchanged and still exist, been solved?
- That’s a good question to which I don’t have an adequate answer. Based on the function descriptions below I would assume that
repairtries to rebuild the broken archive data whilepurge-broken-filesdoes as it says and just removes them.
Assuming the broken data is of file versions that still exist in the source, it would make sense to try and repair the archive - but if the broken data belongs to previous versions then there’s no source data to be used to repair the archive.
My GUESS is that arepairwill do what it can to fix things and if everything can’t be fixed (either due to broken historical versions or some other reason) then apurge-broken-fileswould be in order.
Usage: list-broken-files []
Checks the database for missing data that cause files not not be
restoreable. Files can become unrestoreable if remote data files are defect
or missing. Use the list-broken-files command to see what the
purge-broken-files command will remove.
Usage: purge-broken-files []
Removes all files from the database and remote storage that are no longer
restoreable. Use this operation with caution, and only if you cannot
recover the missing remote files, but want to continue a backup. Even with
missing remote files, it may be possible to restore parts of the files that
will be removed with this command.
Usage: repair []
Tries to repair the backup. If no local db is found or the db is empty, the
db is re-created with data from the storage. If the db is in place but the
remote storage is corrupt, the remote storage gets repaired with local data
(if available).
- As far as I know this has not been addressed, though I’m not quite sure where the problem is. If a block of a file doesn’t change across 5 backup sets then all 5 sets will reference the same block thanks to deduplication. If that block is broken, all 5 sets will inherit the same broken file issue during a restore. If a repair fixes that broken block, then all 5 sets should inherit the same fixed block. (Of course that all assumes
repairworks the way I described above, which is not necessarily correct.)
If you’re referring to the final Feb. 7, 2017 post in that thread then the user is correct - once a block is removed as part of a purge it is as if the block was never backed up in the first place. So the purge is essentially telling the backup set to forget it ever saw that file in the first place.
Hopefully somebody (such as @kenkendk) with a little more familiarity with the repair and purge-broken-file commands will be able to confirm (or correct) my musings…
Ah, I misunderstood that Feb. 7, 2017 post. I thought the listing was of 3 backup sets (several files) and was worried about losing history of other files not in that block. If that’s not the case, then yeah, the behavior you describe is what I had thought. Though obviously losing just the version that is corrupted would be ideal (if repair does this, that would be awesome)
Honestly, I’m not sure if that’s how it does work - it’s how I interpret the description.
I hope to do some tests soon. I plan to try something like the following:
- make test job with a very small volume size (so I don’t have to back up a lot to get multiple files)
- back up enough files to get a few “volumes” (dblock files) in the destination
- run a few more backups (might need to enable the
--upload-unchanged-backupsparameter) - delete (well, rename) an archive known to contain blocks for an “older” backup
- test
repairvspurge-broken-files
So list/purge-broken-files and repair does not do anything. It appears I still need to manually purge by removing the blocks it could not decrypt, as noted in that older ticket. I assume then I’ll follow the rest of those steps: manual purge of the blocks that could not be decrypted, repair, purge, backup?
Additional documentation: list/purge-broken-files says:
No broken filesets found in database, checking for missing remote files
Listing remote folder …
Skipping operation because no files were found to be missing, and no filesets were recorded as broken.
repair says:
Listing remote folder …
Destination and database are synchronized, not making any changes
verify/full-remote-verification and found a number of broken blocks:
Listing remote folder …
Downloading file (47.03 MB) … //vast majority are successful
Downloading file (47.09 MB) …
Operation Get with file duplicati-####.dblock.###.aes attempt 15 of 15 failed with message: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content => Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content
//several of these, which I should manually purge
Just to update on my issue: my server actually had bad ram, verified with memtest86, which was causing these random corrupted blocks. Thanks to duplicati, I may have saved myself some eventual data corruption!
2 Likes
Hello, I had same problem. And this is how I solved it.
Error message during backup:
System.Security.Cryptography.CryptographicException: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content —> SharpAESCrypt.SharpAESCrypt+HashMismatchException: Message has been altered, do not trust content
at SharpAESCrypt.SharpAESCrypt.Read(Byte buffer, Int32 offset, Int32 count)
at Duplicati.Library.Utility.Utility.CopyStream(Stream source, Stream target, Boolean tryRewindSource, Byte buf)
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)
— End of inner exception stack trace —
at Duplicati.Library.Main.AsyncDownloader.AsyncDownloaderEnumerator.AsyncDownloadedFile.get_TempFile()
at Duplicati.Library.Main.Operation.CompactHandler.DoCompact(LocalDeleteDatabase db, Boolean hasVerifiedBackend, IDbTransaction& transaction, BackendManager sharedBackend)
at Duplicati.Library.Main.Operation.DeleteHandler.DoRun(LocalDeleteDatabase db, IDbTransaction& transaction, Boolean hasVerifiedBacked, Boolean forceCompact, BackendManager sharedManager)
at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired(BackendManager backend, Int64 lastVolumeSize)
at Duplicati.Library.Main.Operation.BackupHandler.Run(String sources, IFilter filter)
at Duplicati.Library.Main.Controller.<>c__DisplayClass17_0.b__0(BackupResults result)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
at Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)
Repair - nothing
Verify - nothing
Custom Verify with all parametr and --full-remote-verification
Operation Get with file duplicati-b686628fc3389456aabe4cdd0b9f08726.dblock.zip.aes attempt 1 of 5 failed with message: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content => Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content
Failed to process file duplicati-b686628fc3389456aabe4cdd0b9f08726.dblock.zip.aes => Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content
Purge-broken-files - nothing
At this point I take advantage of upload-verification-file parameter and looked in json file.
I manually verified that hash of this file is different from that in json file.
I did not know which command could help me…
So i manually rename that file > run back > run repair
Repair output
Listing remote folder …
Repair cannot acquire 4 required blocks for volume duplicati-b686628fc3389456aabe4cdd0b9f08726.dblock.zip.aes, which are required by the following filesets:
duplicati-20171212T201736Z.dlist.zip.aes
duplicati-20171215T190000Z.dlist.zip.aes
duplicati-20171216T193650Z.dlist.zip.aes
duplicati-20171217T190000Z.dlist.zip.aes
duplicati-20171218T193850Z.dlist.zip.aes
duplicati-20171219T190000Z.dlist.zip.aes
duplicati-20171220T190000Z.dlist.zip.aes
duplicati-20171222T195536Z.dlist.zip.aes
This may be fixed by deleting the filesets and running repair again
Failed to perform cleanup for missing file: duplicati-b686628fc3389456aabe4cdd0b9f08726.dblock.zip.aes, message: Repair not possible, missing 4 blocks.
If you want to continue working with the database, you can use the “list-broken-files” and “purge-broken-files” commands to purge the missing data from the database and the remote storage. => Repair not possible, missing 4 blocks.
If you want to continue working with the database, you can use the “list-broken-files” and “purge-broken-files” commands to purge the missing data from the database and the remote storage.
Return code: 0
After that I run list-broken files > purge-broken files > and finally backup without error.
Is there simpler way to recover from bad hash error than what I did? Thanks