Backup fails with missing files error - files are present

Support
1

Dear community,

quick summary:

  • I’m using duplicati to backup a folder structure to a remote NAS via WebDav.
  • the connection works, destination system has public IP and correct port-forwarding etc.
  • when I run the backup-job the files are created on the remote system.
  • in duplicati the job fails: “Found 3 files that are missing from the remote storage, please run repair”.
  • the files ARE present on the remote system.
  • I used the CarotDav webdav-client on the machine that’s running duplicati with the same WebDav connection / credentials to verify that. So the remote system should be configured fine I think.

How can I trouble-shoot this? Might this be a bug? I will gladly provide more info if needed to fix this.

Here some more details:

  • client machine with Duplicati 2.0.5.1, Windows 10
  • target: NAS (My Cloud Expert Series EX2 Ultra) connected directly into a Huawei B535 4G LTE Router with public IP.
  • this is the json export of the backup-job: job-config.zip (732 Bytes)
  • I’m using a small test fileset ( <5MB) for testing.
  • I re-installed duplicati several times and tried with creating several new backup-jobs.
  • error message from the log-file (log-level: verbose)
    2020-07-28 16:00:37 +02 - [Profiling-Timer.Begin-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteReader]: Starting - ExecuteReader: SELECT DISTINCT “Name”, “State” FROM “Remotevolume” WHERE “Name” IN (SELECT “Name” FROM “Remotevolume” WHERE “State” IN (“Deleted”, “Deleting”)) AND NOT “State” IN (“Deleted”, “Deleting”)
    2020-07-28 16:00:37 +02 - [Profiling-Timer.Finished-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteReader]: ExecuteReader: SELECT DISTINCT “Name”, “State” FROM “Remotevolume” WHERE “Name” IN (SELECT “Name” FROM “Remotevolume” WHERE “State” IN (“Deleted”, “Deleting”)) AND NOT “State” IN (“Deleted”, “Deleting”) took 0:00:00:00.000
    2020-07-28 16:00:37 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-20200728T131725Z.dlist.zip.aes
    2020-07-28 16:00:37 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-ba48d5b195ff645928f3ae93b98cf7e1b.dblock.zip.aes
    2020-07-28 16:00:37 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i5a201463079f4fbb90d53db1d4f725a5.dindex.zip.aes
    2020-07-28 16:00:37 +02 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteFiles]: Found 3 files that are missing from the remote storage, please run repair
    2020-07-28 16:00:37 +02 - [Profiling-Timer.Finished-Duplicati.Library.Main.Operation.BackupHandler-PreBackupVerify]: PreBackupVerify took 0:00:00:00.098
    2020-07-28 16:00:37 +02 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
    Duplicati.Library.Interface.UserInformationException: Found 3 files that are missing from the remote storage, please run repair
    bei Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
    bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
    bei Duplicati.Library.Main.Operation.BackupHandler.d__20.MoveNext()
    2020-07-28 16:00:37 +02 - [Profiling-Timer.Begin-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteNonQuery]: Starting - ExecuteNonQuery: PRAGMA optimize
    2020-07-28 16:00:37 +02 - [Profiling-Timer.Finished-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteNonQuery]: ExecuteNonQuery: PRAGMA optimize took 0:00:00:00.000
    2020-07-28 16:00:37 +02 - [Profiling-Timer.Finished-Duplicati.Library.Main.Controller-RunBackup]: Running Backup took 0:00:00:00.381

thank you for any thoughts or hints you might have on this!

Kind regards,
oystersauce

2

Welcome to the forum @oystersauce

First step might be to see whether or not Duplicati saw the files in its list request to the WebDAV server.
The Verify files button for the job will show list in Show log → Remote that you can click on to look.

There’s an outside chance that you can scroll back to find history on the 3 offending files. Or can you see basic info such as a timestamp? The files are possibly related to the backup at 2020/07/28 13:17:25 UTC. Checking the backup log from then might reveal something. There are summaries, and a Complete log.

Your log looks like you’re 2 hours ahead of UTC, so I guess that would be about 14:00, so maybe the error referred to the previous backup. depending on how often you run? This is all a guess without enough data.

If you have a log file similar to what you posted that goes back a bit, you can search for the “missing” files, and that will possibly show whether there’s anything odd in their history that might be related to the issues.

There’s some remote file and other history in the database, but looking at databases isn’t everyone’s wish.
You can sort of infer current state by setting upload-verification-file and looking at duplicati-verification.json.

Starting with the simple list look first would seem reasonable. That might affect the entire path for debug.

3

SOLUTION: Time-zone setting on target machine (NAS) was off. Matching it with the source machine resolved the issue.

thank you @ts678 for your answer! you didn’t mention it directly but when you talked about my log beeing two hours ahead of UTC I sensed it might be the time-zone settings on the target machine (the NAS) and yes, that was it. matching time-zone settings on source and target systems did the trick, it works now. thanks again!

1 Like
4

I don’t understand how that worked, but I’m glad it did.