Every couple of days I get the following error “Found files that are missing from the remote storage, please run repair”. Repair appears to fix the problem only to return a day or so later. Several similar topics seem to cover this but they have slightly different issues. The Amazon Drive is only used with Duplicati and I only have one backup configured. I’m backing up about 55GB of data so it’s not particularly large and I have a 100GB Amazon storage allowance.
I’m new to unRAID and Duplicati and am struggling to understand the similar topics. Any help would be greatly appreciated.
I’m running Duplicati - 2.0.3.3_beta_2018-04-02 on unRAID 6.5.3
After running repair, the following message “Destination and database are synchronized, not making any changes” is recorded in the log. This implies the missing files aren’t missing at all.
I suspect unRAID doesn’t have anything to do with the error you’re seeing, but just to be clear are you running Duplicati in a Docker container? If so, do you know if you are using the “duplicati/duplicati” one or the “linuxserver/duplicati” one?
If you look at the Duplicati error details (when viewing the logs in the web GUI click on the error line to get more info) does it include the names of the files that are missing? If so, are they always the same files or are they different?
Historically, issues that sound like yours are often caused by a delay in the destination listing recently uploaded files OR a communications error (the list of files Duplicati gets from the destination gets screwed up while traveling over the network).
As TESTS (not fixes) to see if the errors go away, some things to try include setting options such as:
I can confirm I am running “linuxserver/duplicati” in a Docker container. I’m not sure what the difference is, if any, between the two. The files are different each time:-
The --no-backend-verification parameter DISABLES the pre-backup check to make sure all the files found in the backend are what Duplicati expects. Based on your updated info I doubt this will make a difference - which is good, because disabling this is basically turning off one of the features that helps Duplicati make sure your backups are viable.
The --backup-test-samples parameter tells Duplicati how many filesets (combination of 1 each of a dlist, dindex, and dblock file) to download and test AFTER each backup.
Based on you the dates of the errors with dlist files it sounds like the problem is likely here, My GUESS is that Duplicati is uploading the most recent backup files then once the backup is done turns around and requests a set of files to be tested.
This is normal, but some destinations are a bit slow taking just-uploaded files and making them available for download, if the requested files happen to be the ones that were just uploaded when Duplicati asks for them the destination says “what files?”.
Setting --backup-test-samples=0 says to not test ANY files, so should remove that issue all together IF it’s the actual cause.
Note that this is an issue with the VALIDATION step, not the actual backup - so your backups should be running fine. Also, as more backup archives get added to your destination, it becomes less and less likely that the “random” file check will choose the just-uploaded ones to be tested, so this issue may “go away” on it’s own…
The issue appears to be very intermittent. It took over a week of daily backups before it failed, this time saying 3 files were missing from the destination. I checked Amazon Drive and they were there. Repair fixed the problem. As suggested, I’ve added the following options:-
backup-test-samples=0
no-backend-verification
We will see what happens but why does Duplicate report files missing when they are there?
We haven’t pinned down exactly what is causing this, but my guess is there are a few different reasons (which is probably part of why it’s so hard to figure out).
The meat of the matter is that when Duplicati starts a job it first checks the destination to make sure all expected files are there. It does this by asking the destination for a list of files and compares the returned list to what’s in the local database.
The returned list could differ due to a number of reasons including:
internet screwup (something interrupted or corrupted the list between the destination and Duplicati)
destination screwup (the destination isn’t generating an accurate list - this has happened in the past, but not recently)
provider performance (files recently added to the destination aren’t made available to list / download right away - this should only be an issue if you’re doing VERY frequent backups or possibly as part of the AFTER backup job of verifying specific files)
Duplicati list screwup (Duplicati might be parsing the resulting list of files incorrectly, though if that’s the case the error should happen EVERY time)
Duplicati database screwup (Duplicati’s local database might have gotten corrupted or “forgotten” about some of the destination files)
manual manipulation (a user can manually delete or add files in the destination which can make it go out of sync with the local database)
I suspect the issue will no longer happen with the two parameters you’ve added, but that doesn’t mean it’s fixed. Once you’re satisfied it’s no longer happening I’d suggest removing the --backup-test-samples=0 setting (to let it start testing backups again) and see if the issue stays away.
If it does, that pinpoints the problem to the PRE-backup list check which is actually LESS likely to exhibit this issue than the post-backup verification.
I’ve had to set set the amzcd-consistency-delay to 30 seconds. The default 15s wasn’t enough. I’ll monitor the backups over the next week or so. Thanks for your help.