Hi all. Just got a new MacBook (MacOS Sequoia 15.2) and am setting up Duplicati (fresh install, 2.1.0.2_beta_2024-11-29). Using SSH to backup files to a Samba drive on the LAN. Backup works without errors, and I can see the backup files on the NAS drive. But when I try to restore directly from the backup files I get the following errors:
Failed to connect: An error occurred (In a popup - center screen)
Invalid header value: 00-05-16-07-00 (Bottom of screen, with show/dismiss options. "show" does nothing.)
Searching on the errors gave no useful info. Any ideas what may be happening here, and how to get a successful restore from the backup files?
The restore will write log data into the file, and this should narrow down exactly what file is defective. From there, you can hopefully find an explanation for why this file is no longer in the correct state.
Once you identify the file, you can rename the file so it will not be read by Duplicati, and try again.
2025-01-21 16:59:07 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started
2025-01-21 16:59:07 -06 - [Information-Duplicati.Library.Main.Operation.ListFilesHandler-NoLocalDatabase]: No local database, accessing remote store
2025-01-21 16:59:07 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()
2025-01-21 16:59:08 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (526 bytes)
2025-01-21 16:59:08 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20250121T063000Z.dlist.zip.aes (3.021 MB)
2025-01-21 16:59:08 -06 - [Retry-Duplicati.Library.Main.BackendManager-RetryGet]: Operation Get with file duplicati-20250121T063000Z.dlist.zip.aes attempt 1 of 5 failed with message: Failed to decrypt data (invalid passphrase?): Invalid password or corrupted data
System.Security.Cryptography.CryptographicException: Failed to decrypt data (invalid passphrase?): Invalid password or corrupted data
---> SharpAESCrypt.WrongPasswordException: Invalid password or corrupted data
at SharpAESCrypt.SetupHelper.DecryptBulkEncryptionKeyAndVerifyHMAC(Byte[] data, HeaderEncryptionKey headerKey)
at SharpAESCrypt.DecryptingStreamInternal.ReadEncryptionHeader(Stream stream, SetupHelper helper, String password, DecryptionOptions options)
at SharpAESCrypt.DecryptingStreamInternal..ctor(String password, Stream stream, DecryptionOptions options)
at SharpAESCrypt.DecryptingStream..ctor(String password, Stream stream, DecryptionOptions options)
at Duplicati.Library.Encryption.AESEncryption.Decrypt(Stream input)
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)
--- End of inner exception stack trace ---
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(Stream input, Stream output)
at Duplicati.Library.Encryption.EncryptionBase.Decrypt(String inputfile, String outputfile)
at Duplicati.Library.Main.BackendManager.DoGetFile(FileEntryItem item, IEncryption useDecrypter, CancellationToken cancellationToken)
at Duplicati.Library.Main.BackendManager.DoGetAsync(FileEntryItem item, CancellationToken cancellationToken)
at Duplicati.Library.Utility.Utility.Await(Task task)
at Duplicati.Library.Main.BackendManager.ThreadRun()
So it is clearly crashing due to the inability to decrypt the file. However, I exported the backup configuration (with passwords) to a local JSON file, then copied the passphrase directly from the JSON file and pasted it into the “Restore from backup files” workflow. In theory, using this passphrase from the JSON should allow for decryption, yet it’s not working. To add to the confusion, when I restore using the “Restore from <named backup>” option, it seems to work fine and gives me the directory/file tree to restore from.
The only thing that may (or may not) be related, is that I don’t have signed certificates on my network. When I do the “from files” restore and enter the SSH destination info, I get a “please verify the cert key” warning. I answer “Yes” to that message and the process keeps going. Not sure if/how that may affect what I’m seeing, but I thought I’d mention that in case it makes a difference.
I’m pretty sure I’ve done a “from files” restore in the past, but I can’t swear to that, so I’m not sure if this is a 2.1.0.2 issue.
This is a very different error message than “Invalid header value”?
That would give an error with the transfer, not the password or header error.
Just to ensure you can access your data later, should we discover an issue with the password, make sure to make a copy of the Duplicati-server.sqlite file, as that seems to be able to access the files.
The original error was not related to passwords, it was indeed saying the file was not an AESCrypt encrypted file. The new error is only thrown if the password is somehow incorrect (or the file is damaged in an unlikely way).
Good catch. It is possible that some characters are escaped in the JSON file.
Not ruling it out either, but we do have quite a few automated tests that ensures that this is working as expected.
Could you try to run a “Test” command from the working backup? Click the “Commandline …” button, choose “test” in the command, and remove the contents in “Command line arguments”.
This will run a test to see that all remote files are valid, and at least the first error message indicates something is wrong with the stored files.
So I chased this down a bit, and there may be something to this. Here’s the (partial) passphrase from the GUI (using Show on the general Settings screen):
... {Ak9!HI!4\7CD~@]#U ...
Here’s what’s in the JSON file:
... {Ak9!HI!4\\7CD~@]#U ...
That might explain why restoring form the config file works but restoring directly from the files and manually pasting in the passphrase does not? Does the GUI filter out escapes when you paste in?
No, the GUI does not do anything with the passphrase. It is taken verbatim.
But the export uses the JSON format, which will escape certain characters.
This is not a problem if you import the JSON file again, but when you copy from the file, you need to decode the JSON value back to the string.
As @ts678 mentions, the easiest way is to copy from the configured version, in the UI.
You can also use “Restore from configuration …” which will load in the exported JSON and thus decode the password.
There are a few online tools that supports JsonDecode, but beware they may log the password you send them. Many tools can do it locally, including jq for instance.
Can you try with a bigger sample? Try putting “100” into the “Commandline arguments” to get 100 samples tested.
For the problem with the passphrase, were you able to progress with that?
It is a little problematic to trouble shoot as the actual value is the both a secret and the problem itself.
But the original error seems to have disappeared, or can you still reproduce this?