I am running several test backups, both local and to Amazon and Backblaze B2.
The backup to Amazon Cloud Drive shows every few days an error of 1 file missing. When I go directly to my Amazon Cloud folder online, I can see that the file is NOT missing. What is happening?
Failed: Found 1 files that are missing from the remote storage, please run repair
Details: Duplicati.Library.Interface.UserInformationException: Found 1 files that are missing from the remote storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.String protectedfile) [0x001aa] in <80b1d80627594c8a8e9ee191cb472a3f>:0
at Duplicati.Library.Main.Operation.BackupHandler.PostBackupVerification () [0x00058] in <80b1d80627594c8a8e9ee191cb472a3f>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x008e0] in <80b1d80627594c8a8e9ee191cb472a3f>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass17_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x00036] in <80b1d80627594c8a8e9ee191cb472a3f>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00072] in <80b1d80627594c8a8e9ee191cb472a3f>:0
Does this error happen at the beginning, during, or the end of the backup?
Oh, and is this a small (or newly created) backup such that there aren’t many files in the destination yet?
It is a test backup with just a few files, total of 32MB. I am not sure how to tell when during the backup it happens. The file missing is always one of these kinds duplicati-20180319T010751Z.dlist.zip.aes.
Does the error not happen if you add --no-backend-verification=true to the job?
I ran the backup for two weeks without backend verification and did not get any errors.
As soon as I turn verification back on I do get the errors. I only get them with Amazon Cloud Drive but not with Backblaze B2.
Did you happen to notice if it’s always the same dlist file or does it change each time (implying the “missing” file is just really new and isn’t being listed in the directory yet)?
The missing dlist file changes over time. I think it is just really new, the file name date is one day in the future (on a 9:40 pm PST backup).
Last time there were actually three files missing, a dlist, dblock and index file. Repair worked fine.
This is only happening with Amazon Cloud Drive but not Backblaze B2. In both cases these were only very small test backups.
Ah - that sounds like a timing issue.
Based on the --no-backend-verification=true stopping the errors and the fact that these are small test backups my guess is that Duplicati is uploading a file as it should then moving on to the a test step and because it’s a small backup it might be trying to verify the file that was just upload - but the provider isn’t ready to download it yet.
Can you try using the --backup-test-samples=0 parameter (but leave backend verification enabled) to see if the error still goes away?
--backup-test-samples (default 1)
After a backup is completed, some files are selected for verification on the remote backend. Use this option to change how many. If this value is set to 0 or the option --no-backend-verification is set, no remote files are verified
I did as you suggested but still got the error about missing files.
However, I found in Duplicati, under Backup Destinations - Advanced Options a setting for Amazon Cloud Drive called Consistency Delay.
I think this should fix it based on the description. I selected the default setting of 15s.
Good catch on the --amzcd-consistency-delay parameter, though if the default is 15s that probably means it’s already that, so if you don’t see improvement be sure to try again with a higher number.
--amzcd-consistency-delay
Amazon Cloud drive needs a small delay for results to stay consistent.
Default value: “15s”