Restore: Invalid password or corrupted data AND missing file

Support
#1

I am still testing Duplicati, so have mostly been running backups to different destinations and from different sources. The backups typically give a warning about temporary files it cannot access, but no errors.

However, now I and testing my restore, and this is triggering multiple errors of this kind:

[Error-Duplicati.Library.Main.Operation.RestoreHandler-PatchingFailed]: Failed to patch with remote file: “duplicati-be7f40bdb41784ab2913c20541b76f929.dblock.zip.aes”, message: Failed to decrypt data (invalid passphrase?): Invalid password or corrupted data

My configuration has not changed since I set up my schedules and I have not manually deleted any .aes files.

The restore also has a ‘missing file’ error:

[Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Could not find file ‘C:\Users\Work\Documents\Photo Compare\IMG_8912.JPG’.

This is my real area of concern. I selected this file from the restore pick list, so Duplicati has a record of it, but then it cannot find it in the archive to restore it.

I’d suggest this is not expected behavior, but is it known behavior and is there anything I can do to avoid it?

My Duplicati version is 2.0.5.1_beta_2020-01-18 running on a Windows 10 Pro build 19041.388 with the Windows Feature Experience Pack 120.2202.130.0. There is 32GB of RAM and over 500GB of free disk space, so system resources are not constrained.

#2

One “Invalid password” cause is a browser or extension filling in the wrong password (maybe GUI?), e.g.:

Failed to patch with remote file

I can’t restore from GUI with 2.0.5.1

Technically this error is in the post-restore file integrity test, but that doesn’t rule out a restore problem too.
Does restore to a different folder work? Is there anything acting on the original folder that might change it?

Logging restore at verbose level will show file name and actions. About --> Show log --> Live --> Verbose
What does that have to say about the file, or does it not mention the file at all (which would seem odd…)?

#3

Thanks @ts678, I will look at the browser aspect. It is latest FF and I had already noticed that it offers to populate non-password fields with passwords in Duplicati GUI. Using Edge might get around this as that’s my “dumbed down” browser with no extensions or remembered anything.

With the restore file issue, I was restoring to a different folder and the source file was unchanged across all backup sets. I’ll need to look at the verbose logging and get back to you.

#4

or if you preder Firefox, the below might work, or possibly the below without also doing the history wiping:

Remove a Saved Password from a Browser covers a variety of browsers.

1 Like
Logic and usecase for remote subfolders
#5

I have excluded localhost from the saved password list in FF, but doing the same file restore to the same end point locations gives the same errors.

I have also checked that the .aes file Duplicati is erroring on exists, so it does seem that an error occurred during the save process that has resulted in at least one .aes file not being readable. As I’ve only tried to restore five test files, that suggests a bad outcome for the entire backup set so I have commenced a complete restore to see what’s what.

In case it helps, the verbose error on the .aes file is:

{“ClassName”:“System.Security.Cryptography.CryptographicException”,“Message”:“Failed to decrypt data (invalid passphrase?): Invalid password or corrupted data”,“Data”:null,“InnerException”:{“ClassName”:“SharpAESCrypt.SharpAESCrypt+WrongPasswordException”,“Message”:“Invalid password or corrupted data”,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:" at SharpAESCrypt.SharpAESCrypt.ReadEncryptionHeader(String password, Boolean skipFileSizeCheck)\r\n at SharpAESCrypt.SharpAESCrypt…ctor(String password, Stream stream, OperationMode mode, Boolean skipFileSizeCheck)\r\n at Duplicati.Library.Encryption.AESEncryption.Decrypt(Stream input)\r\n at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:“8\nReadEncryptionHeader\nSharpAESCrypt, Version=1.3.3.0, Culture=neutral, PublicKeyToken=null\nSharpAESCrypt.SharpAESCrypt\nVoid ReadEncryptionHeader(System.String, Boolean)”,“HResult”:-2146233296,“Source”:“SharpAESCrypt”,“WatsonBuckets”:null},“HelpURL”:null,“StackTraceString”:" at Duplicati.Library.Main.AsyncDownloader.AsyncDownloaderEnumerator.AsyncDownloadedFile.get_TempFile()\r\n at Duplicati.Library.Main.Operation.RestoreHandler.DoRun(LocalDatabase dbparent, IFilter filter, RestoreResults result)",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:“8\nget_TempFile\nDuplicati.Library.Main, Version=2.0.5.1, Culture=neutral, PublicKeyToken=null\nDuplicati.Library.Main.AsyncDownloader+AsyncDownloaderEnumerator+AsyncDownloadedFile\nDuplicati.Library.Utility.TempFile get_TempFile()”,“HResult”:-2146233296,“Source”:“Duplicati.Library.Main”,“WatsonBuckets”:null}

Unfortunately, an scheduled backup has overwritten the details on the file that could not be restored, but I expect it’s embedded in the .aes that can’t be read.

Anyway, I’ll keep testing to confirm clean backups before I go ‘into production’ with Duplicati.

#6

Do you have localhost:8200 still there? You want to remove whatever one you use, or test with your Edge.

If you suspect bad files, you can download it and decrypt with AES Crypt or Duplicati SharpAESCrypt.exe.

That should download most files. Another heavy download to test all files is The TEST command for all.
Commandline can do that, but you have to change the command yourself and type all in the arguments.

For best restore testing, use no-local-blocks=true to force Duplicati to obtain all data blocks from remote.

If any of your destination has direct access from Windows or Linux, you could also set upload-verification-file then run one of the DuplicatiVerify scripts in its utility-scripts folder which will check file hashes directly.

#7

Thanks @ts678, all good ideas but I’m going back to basics because it’s clear Duplicati does not operate the way I expect it to.

Doing my complete restore, the NAS fired up and I realized files are being enriched in some manner from the live files, rather than being hoovered out of the archive independently. I stopped the restore and turned off the NAS to try again, and this time files came out without being enriched. This was all done via the GUI, I’ve not broken out the CLI.

But I still get .aes encryption errors and that’s on every one of my trial archives, which are spread across different destinations and from different source combinations. There is no evident reason for this behavior and I’m running out of puff trying to diagnose it. I can’t see a scenario where, having set up my backup jobs via the GUI, some .aes files are ‘corrupt’ such that some can be decrypted while others are okay and can be read.

#8

What is “enriched”? If you mean data is from live files (if possible), see earlier note on no-local-blocks.

Duplicati will attempt to use data from source files to minimize the amount of downloaded data.

But I might have guessed wrong on local blocks. How does an enriched file differ from not-enriched?

Files are not independently stored. A file is stored as blocks, as are updates to a file’s new versions.

The backup process explained
How the backup process works (more details)
How the restore process works (more details)

The set of files to restore are not done one source file at a time, as that would mean downloading a potentially large number of remote files to retrieve a few blocks from each. Instead, remote files are downloaded one at a time, and needed blocks are spread over restored files, written block by block.
After all files are restored, the hash of each file is verifed against original hash to verify its rebuilding.

Have you already made absolutely sure that your browser is not autofilling, as autofill is a great way to produce the independent-of-pretty-much-everything-else problem that you’re seeing? Other ideas too.

I don’t know if I heard this before (I did hear of ones that could not be decrypted). You would have a hard time knowing what files were decrypted without looking at logs in some detail to see downloads working. About --> Show log --> Live --> Retry and seeing successful downloads in a single try would (I think) be evidence that decryption worked. If decryption didn’t work, then I think it would retry up to default 5 times.

If you genuinely have variable decryption problems (as opposed to getting back some source files due to local blocks), then I don’t know why things vary at the remote file level. Still interested in how that’s seen. Duplicati is very resistant to changing passwords on a backup, as that WOULD cause variable issues…

There is also an option to try Commandline or Export As Command-line for The RESTORE command, where any odd password settings would be easier to see because they’d not be so deep in a web page. GUI Commandline needs Edit as text to view the passphrase in the clear instead of shown as stars. Other tests previously mentioned are probably easier than this, but I thought I’d add throw this out too…

#9

Thanks @ts678, I think my terminology is not precise enough in my description. ‘Enriched’ meant that the files in the archive were being compared to files on the network and, as you note, Duplicati used the network ones to minimize the restore effort. That’s not behavior I am accustomed to, normally the restore is independent of the ‘live’ files, esp. when they are being restored to a different directory structure. Perhaps that’s a good GUI option to include, for where users don’t want to do that.

And I understand the block nature of the archives, ‘hoovered’ was merely my offhand description of the restore process extracting files from the archive.

I have disabled the browser autofill, but given your suggestion that my symptoms line up so closely with this issue, I am going to uninstall, delete all the archives, reinstall, and start again. Browsers have become too smart for our own good, sometimes! Though, I do note that many form-based systems do not trigger the browser to attempt an autofill. Is there a method in the code to alert the browser to a non-fill field?

#10

You would have to ask a web developer. Anybody on the forum know?

Whenever I look at this area, it looks messy, and varies by browser…
Another pitfall is that autocomplete and autofill are said to be different.
On the other hand, “special” autocomplete attributes control autofill…

Possible sources:

The autocomplete attribute and login fields

For this reason, many modern browsers do not support autocomplete="off" for login fields

Preventing autofilling with autocomplete=“new-password”

but I’m not sure if the fields that attract autofill to clobber them fit what “new-password” expects.

4.10.18.7 Autofill