I have been for a long time trying to fix this issue, but I have run out of ideas and need help.
It all started where I had to move my Duplicati install to another machine (I use Duplicati to backup to an external “Z:” drive, using version 2.0.3.3_beta_2018-04-02, the database is about ~2gb backing up over 1tb of data).
So I exported the config through the web gui and then imported at the other machine.
Unfortunately while this has happened I had a faulty USB port which “blipped” on and off very quickly, long story short is that I had to repair the disk through Windows.
When I tried to run the imported config on the new machine or the old machine it threw an error:
“Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content”
I tried a repair of the database and got this response:
“The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.”
I tried deleting the database and creating again and again got the same:
“Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content”
Another repair and got this response again:
“The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.”
What I have tried (on both machines):
I have successfully decrypted blocks of the data using the SharepAESEncrypt Tool.
I have tried deleting batches of the last modified blocks of data (restored them back now as it didn’t work).
Repairing the database, resulting in “The database is marked as “in-progress” and may be incomplete.”
Deleted and recreated the database (both through GUI and through Windows).
I have checked the disk again and again to make sure it is not faulty (CHKDSK, SmartMonTools, CrystalDisk, etc).
You moved the install to another machine? I’m assuming the data you were backing up was moved to this machine as well? Otherwise it doesn’t make sense. Try creating a new job manually using the same parameters as the original job. Copy the backed up data to the destination folder, then just to a delete repair job from the gui. It will take a long time to run if you’re DB was around 2GB.
Ah yes, I should of mentioned that the data to backup is on a remote server across the network (SMB).
I tried as you suggested, took a while then came back with:
Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content
I know this to be false as I can decrypt blocks of data manually.
Can you try doing a direct restore from the files themselves to see if Duplicati can decrypt the files. I suspect wherever you’re storing the backup to has slightly modified the content of the files.
If I try a restore, it can list all the files it has, but right at the end it fails and says:
Found 3 files that are missing from the remote storage, please run repair
A repair then results in:
The database was attempted repaired, but the repair did not complete. This database may be incomplete and the repair process is not allowed to alter remote files as that could result in data loss.
Can you list broken files and see if anything comes up. If you can browse your files it means you can decrypt. What may have happened is a discrepancy between what files are located in the storage location and what files are recorded in the database.
Listing broken files returns:
Cannot continue because the database is marked as being under repair, but does not have broken files.
Return code: 100
If I then delete the database and recreate, then it says this:
Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content
Listing the broken files again returns the same:
Cannot continue because the database is marked as being under repair, but does not have broken files.
Return code: 100
I’ve looked at other backup programs and they dont offer what Duplicati has, so I really want to be able to fix this otherwise I will have to use another program for reliability reasons if I cant recover from this.
Thanks for that. I’m really out of ideas of what could cause this. As a last step I would attempt to recover from the command line. Even when the DB is out of sync the data is recoverable using the “Duplicati.CommandLine.RecoveryTool.exe”. However you’ll probably need to start your backup cycle again which means another full
Thanks for your help all the same.
Is it any help if I provide any logs? Not 100% sure on the correct way to do this mind you.
I have other backups that I can use so there is no data loss for me, I am just more concerned how easily it was to break my backup set even though the data is intact (or close enough).
When you tried your test restore did you do it through the imported (and possibly corrupt) job or did you use the main menu “Restore” option with “Direct restore from backup files …” or “Restore from configuration …”?
Since it sounds like you’ve already directly verified your password against the .aes destination files my guess is you will be able to do a “Direct restore”.
If you can confirm that then we’ve at least verified the existing backup is still valid and that it’s likely an issue with the (possibly corrupted) imported that’s causing the issues.
Well the Recreate completed, and consistency checks completed, marking database as complete implies the database has been recreated so I’m assuming there are no more “database was attempted repaired, but the repair did not complete” issues.
Your “registering a missing remote file” messages mean the database expected some dblock files on your USB drive but they weren’t included in the list of files found on the drive.
Now that they’re flagged as missing I suspect a database REPAIR (not a recreate) should make those errors go away, but it would be good to first manually check the files really aren’t there - can you check if they’re on the USB drive?
I don’t think Duplicati is to blame. If the back end storage experiences failure or corruption, it would break any backup system. If you want to protect yourself from this type of failure you’d need to use a better storage system. Cloud based providers are more resilient, or you could target a NAS with redundant RAID, for instance.
I can confirm that those files are not on the USB drive.
I tried a repair and it resulted in the following error:
The database was attempted repaired, but the repair did not complete. This database may be incomplete and the repair process is not allowed to alter remote files as that could result in data loss.
I do not understand why its saying it is not allowed to alter remote files? Why would it need to? I assume by remote it means on the backup and not the source?
Similar error if I just try and run the backup (in regards to your question asking if I still get the “Failed to decrypt data” errors):
The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.
Thanks for the suggestion of Cloud, NAS, or RAID. For me I cant justify either the additional cost or in the case of Cloud it would be easier to just go to a full cloud backup solution instead (and I would rather stick to local anyway).
In my experience I have never had a backup system fail because the backed up data (not the backup mechanism) has failed in part.
This is why I have concern if a single file loss can result in the whole backup being feasibly unrecoverable (there are over 40,000 file blocks in the backup location).
I really like the idea of Duplicati, so I really hope I can fix the issues and continue using it.
Unless every single one of your backed up files has a data block in the one missing file, the backup shouldn’t be considered unrecoverable.
Though I suppose we should clarify what is meant by “unrecoverable”.
Duplicati is able to restore any / everything it can find at your destination - even if you have no local database / backup job and/or if remote files are missing. In THAT sense, the backup is not “unrecoverable”.
What Duplicati won’t do (unless you force it) is to continue adding versions to a backup destination that is already known to have an underlying problem since allowing that to happen potentially puts future versions at risk if they rely on the existing / known bad blocks.
So in the “can’t continue backup up” sense, yes - the backup is “unrecoverable” until it’s repaired, purge, compacted, etc. Those sorts of tasks should normally get you out of the issues you’re having (making the backup usable as a destination again), so I’m not sure why it’s not working.
In theory the “worst” case scenario would be that you have to declare this current destination orphaned as a backend (but still usable for restores) and start a new job to a new destination. Once old enough to be retired the current “broken” destination could be deleted. But again, that shouldn’t have to happen so it would be great if we could figure out the underlying issue.
@kenkendk, do you think “Create bug report …” might provide the right info to figure out what’s going on with the “Invalid header marker” warnings and “registering a missing remote file” errors?
Disclaimer: If you lose your encryption passphrase then you’re pretty much stuck, but hopefully that should be fairly obvious to most people.
The “registering missing remote file” message shows up, when Duplicati finds a dindex file that references a missing dblock file. I managed to create a the error in testing by messing up the deletion of dindex files. Unfortunately, Duplicati currently registers the blocks with whatever index file first mentions it. I have written some code that swaps blocks around to better match the files that are actually found, as opposed to stupidly insisting on using the non-existing files. It appears to work, but there is a related error that I have not yet fixed.
For the “invalid header marker”, it would be sufficient to open the file with a hex editor and look at the first 8 bytes or so. The first 3 bytes should be AES, if they are just random values, I don’t know what happened there.
Ufortunately, I do not currently have time to look through bug reports. But maybe it would make sense to build a small analysis tool (callable from the WebUI) that looks through the RemoteOperation table and produces some information, like:
file 123.zip is no longer present remotely. It was created on 2018-01-10, and present until 2018-03-04.
The operation performed before this date was a backup operation (2018-03-03) and the operation after was a compact (2018-03-05).
Looks like I spoke too soon below. Recreating the database works for one backup. Any subsequent backups will result in the same error again. I have tried this twice now.
Just wanted to update this in case my experience can help anyone in the future. I’ve been getting some backup failures for a while based on either of the following two errors:
Error: A WebException with status NameResolutionFailure was thrown
, or
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
I haven’t looked closely to see what is causing these issues, but it’s likely just me having my laptop off during the normal scheduled backup and then taking into work and Duplicati automatically starting behind my work firewall.
My laptop has been getting a blue screen of death (Windows 10) lately. I was getting ready to run some hardware tests so I did a manual backup once more. That’s when I noticed Invalid Passphrase issue had recently started. I don’t see anything new that caused it to start in the logs. Assuming the BSODs are maybe to blame for stopping it mid backup or something. I was not able to complete a new backup, or recover any files, or repair the database. I was using version 2.0.3.3. The backups would run a long while before finally throwing the error at the “deleting unwanted files” stage.
So I ran a Recreate of the profile database which completed successfully. I confirmed I could manually decrypt the files using AESCrypt for Windows. I still got a Warning “Found 1 files that are missing from the remote storage, please run repair” when trying to restore a file. So I ran a database Repair this time, and it completed successfully saying there was nothing out of sync. Now I’m able to restore my files again and I’m not getting any errors/warnings on the profile.
I’m finally getting successful backups again on the profile and everything appears to be working.
