One of our clients we use Duplicati for as a service install keeps getting this error. Have tried repair and recreate/repair on the database several times but it keeps popping back up. Is there a way to set it to ignore the “extra” remote files?
Log below.
Failed: Found 4 remote files that are not recorded in local storage, please run repair
Details: Duplicati.Library.Interface.RemoteListVerificationException: Found 4 remote files that are not recorded in local storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable1 protectedFiles) at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action1 method)
Log data:
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-b140f69b7df3e42f09883ac7ebf16dda3.dblock.zip
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-b1b3d94ac2e2b4545b8ad6d2c26a7af11.dblock.zip
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-b640876824aa54f61b0e28ad804c8902b.dblock.zip
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-becad0297223e4393abc3d8e625fb8418.dblock.zip
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-ibab1b04dd17b4033bbaf7a8dcbc96fe4.dindex.zip
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-ibeb44d5e77a848c6bef627a63a10d97e.dindex.zip
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-id12281d79ba2468698921271efd9d373.dindex.zip
2024-07-22 22:00:19 -05 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i04f82a5d11314183abcb244c275decb2.dindex.zip
2024-07-22 22:00:19 -05 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraRemoteFiles]: Found 4 remote files that are not recorded in local storage, please run repair
2024-07-22 22:00:19 -05 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.RemoteListVerificationException: Found 4 remote files that are not recorded in local storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable`1 protectedFiles)
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.d__20.MoveNext()
The error message here indicates that the remote folder contains files from another backup. It is usually caused by two backups pointing to the same folder, which will not work. If this is the case, make sure to use different folders for the remote destination on each backup.
It can also happen if you make a test backup to the destination, and then delete the backup. The remote destination will then have these files already. If this is the case, manually delete the remote files or choose another remote folder as the destination.
There is not an option to ignore the files, because the restore will behave unexpectedly with additional files.
What’s hard to explain is that there are not only extra dblock files, but missing dindex files too, and no mismatch on dlist at all, although possibly the backup got interrupted before dlist upload at end.
Presumably meaning this class of error, and not the exact same files (which would be even odder).
It sounds like maybe there are multiple clients. If so, what is this one doing that might be different?
If somehow the explanations given fit, great, otherwise how awkward is it to get more debug data?
It seems a little surprising to have so few files in a computer backup. Can you see the destination?
The Complete log in the job log might have some useful information, and more logs could be set.
Never store multiple backups in the same remote location. Backup data files from multiple backup jobs will interfere with each other and will result in failed backup operations. Always create a new remote folder for each backup job that you set up.
Regarding the small number of files, it might be normal (as updates to a backup can be small, however you might want to make sure Remote volume size on Options screen is near 50 MB default, because one common error is to set this way too large inadvertently, causing troubles.
If none of the above apply, we will probably need a Complete log from failure and run before. Sometimes failures will not leave the usual log, but it can be significant if that’s known. There’s About → Show log → Stored that may show prior errors from a failure leading up to mismatch.
If it is a rare problem without known interference as described above, it may need extra logging. log-file=<path> with log-file-log-level=Retry would be a good start, but heavier logs are possible.
Thank you for the replies, sorry about my own late reply here.
@kenkendk Yeah, this is the only backup job that’s ever run for this computer in duplicati, so neither of those should apply I’m pretty sure?
@ts678 They’re pretty small file-level backups but we have different buckets set up for our different clients – we do IT management for small businesses like landscaping companies, etc, and they usually don’t have a ton to back up – we’re just extra diligent about it. No other backups or clients or anything point to this bucket though. It’s a Backblaze B2 bucket fwiw, set to smart backup retention, and we haven’t encountered this error with anyone else.
Troublingly, I’ve done a complete recreate of the database several times but I can check next time I’m on it to see if I can do it again. The Remote volume size is one I might have to check, as I believe we may have set that higher than 50MB. I’ll check those, and return with detailed logs if necessary.