Hi thanks for all the work on this. It looks like just want I’m looking for - a way to backup my PC to my cloud storage. I’m using Amazon Cloudrive and I’ve managed to get an Auth code and succesfully tested it. However when I tried a test backup I get a red message at the bottom of the screen:
Looking at the log it seems to be looking for three files:
Fatal error
Duplicati.Library.Interface.UserInformationException: Found 3 files that are missing from the remote storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)
Can any-one please give me any pointers as to what I need to do to fix this? How do I “run repair”? Thanks,
This is interesting, as I am having the same problem.
I am also new to Duplicati (another Crashplan refugee) and I’m experiencing problems backing up to Amazon Cloud Drive as well. As a test I’ve set up two backup sets on the same data. One backup goes to a local network drive, the other goes to my Amazon Drive account. The local backup is working fine, but when I backup to Amazon Drive sometimes it works and sometimes I get the same “Files are missing” error that Prospector is getting. Here are the error messages:
To try to isolate the problem, I created a third backup set using the exact same parameters and source file sets, only this time backing up to a Google Drive. The GDrive backup runs without error every time (as does the local backup). File adds, deletes, and repeated runs all work fine.
Is there a problem with the way Duplicati interacts with Amazon Cloud drive that might be causing these errors?
Yes. Amazon Cloud Drive is “eventually consistent”, so what can happen is that you upload the file, but when you list what files are present, it is not yet updated.
You can use the option --amzcd-consistency-delay=60s to make Duplicati wait 60 seconds before trying to list files. By default it waits 15 seconds.
I tried delay=60 and am still getting errors. I’ve been increasing the timeout value to see if it works better. I’m currently up to 180 seconds and still getting errors. I’ll keep testing, but at some point I’m going to have to try something else. I’ll let you know how it goes.
BTW: A good way of showing your appreciation for a post is to like it: just press the button under the post.
If you asked the original question, you can also mark an answer as the accepted answer which solved your problem using the tick-box button you see under each reply.
All of this also helps the forum software distinguish interesting from less interesting posts when compiling summary emails.
So I upped the delay to 240 seconds and was still getting intermittent errors. I’ve since moved on to Backblaze B2, which turns out to be cheaper than Amazon anyway. If someone wants to follow up on this I’d be happy to serve as a test case.
I believe I’m facing the same issue on Amazon cloud drive. I’m using Duplicati - 2.0.2.12_canary_2017-10-20 on Mint 18.2 with latest mono. This is is driving me nuts as repairing does not fix it. Even recreating the database appears useless. Even a small files set such /etc is enough to trigger the problem after some time.
I did not touch any option or delay yet.
Thanks Kenkendk, I added the option and will let it run for some time. It seems already better but I think it may be linked to the fact the set is very small (only /etc) and is backed up very quickly.
It still happens despite --amzcd-consistency-delay=60s though less often than before. and the backup set “/etc” is only 30MB. Larger sets (on other computers) seems unaffected or even less affected.
Pretty much in the same boat here - trying to test Duplicati with ACD as the target. Whilst Backblaze B2 seems to be an interesting alternative, I prefer ACD, as am getting 3x upload speed with it maxing out my bandwidth (not trying to say B2 is bad, it’s most likely due to my location).
I’m using Duplicati - 2.0.2.12_canary_2017-10-20 on Windows 10 64-bit, test backup set is ~50MB (though I tried bigger / smaller ones as well),
This is what happens when backing up to ACD:
1st run of backup job throws an error: Found 1 files that are missing from the remote storage, please run repair
Running repair and getting this: Listing remote folder … Destination and database are synchronized, not making any changes Return code: 0
However, on the home page backup status still says: Last successful run: Never
Though I can restore the files from restore option anyway.
After running the same backup job again with no files changed / added - last successful run is properly displayed
3rd run, some files added - no errors, 1st and 3rd jobs properly displayed.
4th run, 4 files deleted and some added on the source - error stating this: Found 2 files that are missing from the remote storage, please run repair
"Broken" version not shown on the home page, but again it is available for restores though
Running repair again with the same effect - no changes to job status (4th run not visible)
Running another “empty” job (no files changed/added/deleted) fixes it again.
So in general all backups actually work, but false positive errors are generated during baseline and whenever a file is deleted on source.
Can any of this be fixed? For the deleted files - is there any flag to set? With versioning I’d say this is obvious one wants to keep deleted files in backup images.
@radek, what value are you using for the --amzcd-consistency-dela parameter as mentioned here?
@kenkendk, if I’m understanding correctly this seem to be happening because the ACD file list is somehow being cached / delayed. Does a file verification happen at the end of a job? If there a way to disable the end-of-job verification so only the begging-of-job check happens (hopefully after the ACD cache is all caught up)?
Thanks for the verification. I’m not an expert in ACD but It’s sounding like an oddity in how their system works so I’m not sure exactly how to address the problem (other than the suggestion I made to kenkendk in my previous post).
Just out of curiosity, does enabling --no-backend-verification help at all? (This isn’t a fix so much as a test…)
The problem seems to have “disappeared” for me. But I now have many versions of /etc on AZCD, Il will restart with a “fresh” back and see what happens.
button under the post.