"found <n> files that are missing from the remote storage" investigated a reason for unsuccessful backup


#1

For some months I was struggling with unsuccessful backups on a remote server (1&1, WebDAV).
The backup always seemed to run fine, the files are created as expected, but at the end an error log is displayed telling that n files are missing from the remote storage. The files listed as missing are identical to the backup files that have been created.

I observed this effect on different computers. The settings for the remote backup to this kind of servers have been specified as documented. Login data is ok and the remote connection tests runs fine too.

Now i found out that it occurs only if the target folder (‘Path on server’) is not placed on the top level of the WebDAB server account but at least one level lower, e.g. ‘Path on server’ = “MyBackup\Level2”.
Backing up to a top level folder, e.g. ‘Path on server’ = “MyBackup”, works fine.

I could verify this with different target folder settings.

I would be interested in your investigations on this. Maybe it is a general issue with similar remote server configurations and this information can help solving issues reported by other users before.

Thanks in advance,
Boris


#2

Hi, @Boris - welcome to the forum!

Am I correct that 1and1.com is a domain registrar / host not a dedicated cloud drive?

I believe Duplicati does a pre-backup file validation where it compares what it thinks should be at the destination with what is actually there. If the missing files error doesn’t happen at that time but only at the end of the backup then my guess is Duplicati is doing a post-backup “what files do you have” check faster than the WebDAV destination is returning them in the file list.

If that’s the case, then I would expect all your backups are working. Have you tried a restore test?


#3

Am I correct that 1and1.com is a domain registrar / host not a dedicated cloud drive?

It is a german dsl service provider, domain registrar, … and they do provide cloud storage too. There are some restrictions for the number of files (overall, per folder), but this is not critical for my backups.

I believe Duplicati does a pre-backup file validation where it compares what it thinks should be at the destination with what is actually there. If the missing files error doesn’t happen at that time but only at the end of the backup then my guess is Duplicati is doing a post-backup “what files do you have” check faster than the WebDAV destination is returning them in the file list.
If that’s the case, then I would expect all your backups are working. Have you tried a restore test?

No i haven’t tried yet because even a database repair nor a 2nd backup run did never succeed. But i may check if restore is working.


#4

I’d suggest a quick restore test just to make sure uploads are working correctly. If that fails then there’s a bigger issue going on.

But if it works then it’s likely a smaller issue that the validation step needs a delay or retry option to better handle slower destinations. Again, if this really is what’s going on then it’s not a great way to run as you might actually have an upload failure but I would expect the worst case would be that you’d find out during the beginning of backup destination check that the previous backup hadn’t finished correctly.


#5

A ‘Restore’ brings up an empty dialog:

A database ‘Verify’ results in the following message:
“remote storage, please run repair bei Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile) bei Duplicati.Library.Main.Operation.TestHandler.Run(Int64 samples) bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method) bei Duplicati.Library.Main.Controller.Test(Int64 samples) bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)”

‘Repair’ does also nothing:
“scheduling missing file for deletion, currently listed as Uploading: duplicati-20171225T161211Z.dlist.zip.aes”

Again: this happens only with target folders on a sublevel. Toplevel folders work fine. I did a 100GB backup last night on this WebDAV server using a toplevel target folder.


#6

I had the same issue with the 1+1 servers but I could identify the root cause for that behavior: I have used the Windows Path separator ("\") for the destination directory.

Switching to UNIX path separators ("/") fixed my issues and it worked fine afterwards.


Found 114 files that are missing from the remote storage, please run repair