Restore operation failed. Failed to retrieve file xxx.dblock.zip.aes & Message has been altered, do not trust content

Support
1

Hello,

Today, I tried to restore some photos. However, to my surprise, I couldn’t restore over 106 photos approximately.

The backup source was from a Linux OS, to a Windows OS via SMB. I did a successful restoration test on my Linux OS before going ahead and deleting the Linux OS.

Now, I tried to restore from the Windows OS but returned me the following errors:

“RestoredFiles”: 862,
“SizeOfRestoredFiles”: 1622084291,
“RestoredFolders”: 0,
“RestoredSymlinks”: 0,
“PatchedFiles”: 0,
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“DeletedSymlinks”: 0,
“MainOperation”: “Restore”,
“RecreateDatabaseResults”: null,
“ParsedResult”: “Error”,
“Interrupted”: false,
“Version”: “2.0.8.1 (2.0.8.1_beta_2024-05-07)”,
“EndTime”: “2024-06-08T18:47:10.5965247Z”,
“BeginTime”: “2024-06-08T18:44:43.8138472Z”,
“Duration”: “00:02:26.7826775”,
“MessagesActualLength”: 74,
“WarningsActualLength”: 1,
“ErrorsActualLength”: 106,
…
“Errors”: [
“2024-06-08 21:46:01 +03 - [Error-Duplicati.Library.Main.AsyncDownloader-FailedToRetrieveFile]: Failed to retrieve file duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip.aes\r\nCryptographicException: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content”,
“2024-06-08 21:46:01 +03 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-PatchingFailed]: Failed to patch with remote file: "duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip.aes", message: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content\r\nCryptographicException: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content”,
“2024-06-08 21:46:52 +03 - [Error-Duplicati.Library.Main.AsyncDownloader-FailedToRetrieveFile]: Failed to retrieve file duplicati-b46c98e107459437fbdb2abb374d089a5.dblock.zip.aes\r\nCryptographicException: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content”,
“2024-06-08 21:46:52 +03 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-PatchingFailed]: Failed to patch with remote file: "duplicati-b46c98e107459437fbdb2abb374d089a5.dblock.zip.aes", message: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content\r\nCryptographicException: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content”,

The rest of the errors were like:

[Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "filename". Error message was: Could not find file

The Duplicati GUI do list the missing photos, so I tried restoring them one by one instead of selecting the whole folder. Didn’t work.

I tried using Duplicati.CommandLine.exe with the following commands: repair | list-broken-files, yet none of them returned any error, which indicates nothing were wrong with the dblock files. I even removed these dblocks intentionally, and only then the errors showed up which firmly backup -no pun intended- the previous statement.

I tried using SharpAESCrypt.exe and downloaded AESCrypt.exe to decrypt these dblocks manually. Both of the software’s returned:

Message has been altered, do not trust content

I entered a wrong version of my passphrase too, and the software’s told me that the passphrase was wrong, which means that the dblocks are decryptable with my current, correct, noted-down passphrase.

I did try other methods and read official forms from AES Crypt, but can’t remember right now, as I am so stressed as the photos are important to me.

I really don’t understand, the restoration process was totally successful before I deleted the original location (on the Linux OS), and no backup job were run after that, nor I didn’t close the Duplicati software inappropriately.

2

Unfortunately, the restore on the same machine has some “optimization” to use already existing files. This is enabled by default and hidden from the user, and it has been discussed that it is dangerous because people will assume everything works, until the computer dies and it doesn’t. It can’t be expected for you to know this.

You seem to already have tried everything I would suggest with the broken files, so I doubt there will be a way to recover that. The best bet you have is to hope that the necessary parts of the Linux partition were not overwritten yet and use data recovery on that drive. I am not an expert on that, but you shouldn’t turn on that computer until you tried everything in that regard. There are some programs that can scan for deleted photos, you should try those.

1 Like
3

Ahem

dd if=/dev/urandom of=/dev/theLinuxDrive bs=4096 status=progress

XD

4

That is, so, so ,so, so, soooooo bad guys :eyes:. Now I have to deal with over 12 years of lost happy moments.

5

Even though this is a stupid question: Do you have any older full image backups of that system lying around anywhere, or SD cards with those pictures, anything?

You can look at the binary data of the encrypted files in a hex editor, or even with a text editor. It should start with AES...CREATED_BY. If that is there, maybe only part of the files are broken. But I have seen cases where just a file full of zeros was created for an unknown and unreproducible reason.

1 Like
6

Unfortunately, I don’t do any full image backups. I only do backup specific files, including the photos in question, as I am limited with my storage size, so the less the better.

OH listen, I may have them on a USB flash. The photos were on them before I moved them to my Linux disk. Funny enough, I argued with my mom to not keep copies laying around as no need for them anymore XD.

Hopefully, I will find them and nothing be missing. I will update you guys for more attempts.

7

That is what I saw:

AES !CREATED_BY SharpAESCrypt v1.3.3.0

Also, looking into the file, no random zeros were found. Hopefully only few broken files are there as you mentioned. But how can I reconstruct them? I tricked AES Crypt software into keeping the decrypted zip file/dblock, instead of removing it upon existing with the error of:

Message has been altered, do not trust content

I can extract the contents of the zip too. It is a bunch of random named files, each size of 100 KB.

8

The fact that you are able to open the zip files is very good news. The files with random names and 100 KB size is what you would expect, because the files are broken up into blocks for deduplication. You can probably get it restored with the recovery tool:

Depending on how much space you have, the easiest way would be to follow the steps in the manual starting with download. When you get the decryption errors again, manually add the files that you decrypted by ignoring the check and make sure the name stays the same, just remove the .aes.

If you don’t have space to store the whole backup again, we can probably figure out how to only restore the broken files, but that will be more complicated.

1 Like
9

I have made a second copy on the current machine of the Duplicati backup files that I was experimenting on to try and recover the files.

I will go ahead and read through the guide. Hopefully I will be able to restore the files successfully. I will update you of any problem I face.

I really appreciate your time and efforts :kissing_heart: (no homo :eyes:)

10

I downloaded the files. The same dblock files that were in the error log showed up:

93: duplicati-b46c98e107459437fbdb2abb374d089a5.dblock.zip.aes - downloading (49.91 MB)… - decrypting … error: System.Security.Cryptography.CryptographicException: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content —> SharpAESCrypt.SharpAESCrypt+HashMismatchException: Message has been altered, do not trust content
at SharpAESCrypt.SharpAESCrypt.Read(Byte buffer, Int32 offset, Int32 count)
at Duplicati.Library.Utility.Utility.CopyStream(Stream source, Stream target, Boolean tryRewindSource, Byte buf)
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)
— End of inner exception stack trace —
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(String inputfile, String outputfile)
at Duplicati.CommandLine.RecoveryTool.Download.Run(List1 args, Dictionary2 options, IFilter filter)

105: duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip.aes - downloading (49.93 MB)… - decrypting … error: System.Security.Cryptography.CryptographicException: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content —> SharpAESCrypt.SharpAESCrypt+HashMismatchException: Message has been altered, do not trust content
at SharpAESCrypt.SharpAESCrypt.Read(Byte buffer, Int32 offset, Int32 count)
at Duplicati.Library.Utility.Utility.CopyStream(Stream source, Stream target, Boolean tryRewindSource, Byte buf)
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)
— End of inner exception stack trace —
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(String inputfile, String outputfile)
at Duplicati.CommandLine.RecoveryTool.Download.Run(List1 args, Dictionary2 options, IFilter filter)

I decrypted these two dblocks manually as you said, and added them to the download folder.

I then created the index file according to:
https://docs.duplicati.com/en/latest/08-disaster-recovery/#creating-an-index-of-downloaded-files-using-the-recovery-tool

I got another error for the same two dblocks:

91: E:\TryRestore\duplicati-b46c98e107459437fbdb2abb374d089a5.dblock.zip error: System.NullReferenceException: Object reference not set to an instance of an object.
at SharpCompress.Readers.AbstractReader2.get_Entry() at SharpCompress.Readers.AbstractReader2.LoadStreamForReading(Stream stream)
at Duplicati.Library.Compression.FileArchiveZip.LoadEntryTable()
at Duplicati.Library.Compression.FileArchiveZip.ListFilesWithSize(String prefix)
at Duplicati.Library.Compression.FileArchiveZip.ListFiles(String prefix)
at Duplicati.CommandLine.RecoveryTool.Index.Run(List1 args, Dictionary2 options, IFilter filter)

103: E:\TryRestore\duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip error: System.NullReferenceException: Object reference not set to an instance of an object.
at SharpCompress.Readers.AbstractReader2.get_Entry() at SharpCompress.Readers.AbstractReader2.LoadStreamForReading(Stream stream)
at Duplicati.Library.Compression.FileArchiveZip.LoadEntryTable()
at Duplicati.Library.Compression.FileArchiveZip.ListFilesWithSize(String prefix)
at Duplicati.Library.Compression.FileArchiveZip.ListFiles(String prefix)
at Duplicati.CommandLine.RecoveryTool.Index.Run(List1 args, Dictionary2 options, IFilter filter)

I don’t think that sounds good. Right?

11

It has some issues with the zip file headers. What did you use to extract them before? Maybe you can create new zip files with the extracted contents and use that?

1 Like
12

I used 7-zip.

Windows built-in zip function didn’t work. It returned this error:

Cannot Complete the Compressed (zipped) Folders Extraction Wizard

The compressed (zipped) folder is empty

Before you can extract files, you must copy files to this compressed (zipped) folder.

7-zip returned an error too, but didn’t seem to affect the extraction or the count of files at least. For example:

Headings Error
Unconfirmed state of archive
Warnings:
There are some data after the end of the payload data
Data error : iGR00gDt9bPg68UD7th9sGGEI5uxSXzCCW_gYrTqlCc=

I will go ahead and zip them into a new zip file with the same naming of the original files.

13

Wow, it seems like it really listed my lost photos and their original sizes I guess.

I won’t be optimistic though.

I am going to start the restore process according to:
https://docs.duplicati.com/en/latest/08-disaster-recovery/#restoring-files-using-the-recovery-tool

Wish me luck :crossed_fingers:.

Edit: @Jojo-1000

It showed me the following error:

Using set 0 with timestamp 5/29/2024 9:55:16 AM
Building lookup table for file hashes
Program crashed:
System.Exception: Unexpected long line starting at offset 11063131, read 128 bytes without a newline
at Duplicati.CommandLine.RecoveryTool.Restore.HashLookupHelper.ReadNextLine()
at Duplicati.CommandLine.RecoveryTool.Restore.HashLookupHelper.d__11.MoveNext()
at Duplicati.CommandLine.RecoveryTool.Restore.HashLookupHelper…ctor(String indexfile, CompressedFileMRUCache cache, Int32 blocksize, Int32 hashsize)
at Duplicati.CommandLine.RecoveryTool.Restore.Run(List1 args, Dictionary2 options, IFilter filter)
at Duplicati.CommandLine.RecoveryTool.Program.RealMain(String _args)

Edit 2:

I think I may found the problem. Inside the index.txt, there are long strings for the same two dblock files that almost extend for over 160 lines for each dblock as opposed to one hash per dblock.

Example:

f146008c29e418bb803d047c28e542c.dblock/iE6pWpbhq93dSudyWKvsIlt7dPwMc2N61CGrWHyqiRE=, duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip

duplicati+b4f146008c29e418bb803d047c28e542c.dblock/iGR00gDt9bPg68UD7th9sGGEI5uxSXzCCW/gYrTqlCc=, duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip

duplicati+b4f146008c29e418bb803d047c28e542c.dblock/iKBtwqZw0Nn+VY2ldis4H02LxusZQG06aepF3VbTQsA=, duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip

duplicati+b4f146008c29e418bb803d047c28e542c.dblock/iTLrj4rUU/qy3oBhUhSsXITJZpjyT5nMDHqNxHY2v5M=, duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip

duplicati+b4f146008c29e418bb803d047c28e542c.dblock/iTmtGhDIRcTShccK8jh0vkNOwnAhDaS/kCVP61zrSGU=, duplicati-b4f146008c29e418bb803d047c28e542c.dblock.zip

14

Did your re-zipping add another folder layer inside the zip file? The files should all be directly in the root of the zip file.

1 Like
15

Oh yeah indeed. Sorry I didn’t notice that :slight_smile:

Edit:

Still showed some errors, but hopefully not much of a problem.

16

@Jojo-1000

I was able to restore a lot of lost photos :partying_face::clinking_glasses:.

The count of still lost photos:

Folder A: 9
Folder A\B: 4

Total: 13

However, I am not sure whether some are corrupt or not. Everything seems to have a generated thumbnail, which could means all good, at least for photos.
As for videos, I think I will have to watch all one by one to make sure no parts were damaged.

Either way, I will check for the other copies from the other storage devices.

Again @Jojo-1000, thank you very very very much for what you have done to help me through this :handshake:.

Honestly, I am not going to use any backup software anymore, especially Duplicati. lol

PS: I will be posting a documentation for the steps we went through later today.

17

Hm that sounds interesting. Is there a way to disable that?

18

no-local-blocks

Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.

Good practices for well-maintained backups suggests using no-local-blocks for test restore, but the way to have spotted the corrupted encrypted file would have been The TEST command which is similar to default Verifying backend files except for setting the number of sample sets to test, and ability to also unzip the file.

Duplicati always looks at the Destination for file presence and length. but internal damage is harder to spot, because typically it requires actually downloading the files, but sometimes this is slow and expensive to do.

backup-test-samples defaults to 1, so is a very light sample of a large backup.
backup-test-percentage was added to allow better coverage of such backups.

Backup Test block selection logic explains how it tracks the destination testing.
I’m not sure if there’s an option to make sure everything is tested at least once.

upload-verification-file can be used for OS-readable destination, such as yours.
You would run utility-scripts/DuplicatiVerify.py to verify all of the destination files.

Note that this (and many tests that avoid downloads) test against the database.

19

Thanks alot, super valuable information.

1 Like
20

The intention of this was to save time when restoring, by not downloading the blocks if they are already on the system. But it is quite bad if you test the restore with the flag enabled.

There are two headers in a zip file, so it can survive some corruption. But the fact that you see corruption inside the zip files, is likely an indication that the files were truly damaged, which is what trips the decryption to give the “content has been altered” message.

It is a bit strange that this damage was not discovered by the test process that is part of the backup. Essentially a random sample of files are downloaded and their hash value compared to the value recorded at the time of generation. If these differ you will see a warning.

I would be really concerned if the files were damaged locally (by Duplicati) before being uploaded, as that would not be discovered until the data was needed.