We often get below error report or the error saying
"Details: Duplicati.Library.Interface.UserInformationException: Found 1 files that are missing from the remote storage, please run repair…"
from Duplicati, this happens for almost all the users.
Running repair doesn’t help always,It works only once in a while. Even deleting the backup & create a new one will work for some days then again
it throws the same error after a week or so.
Thanks very much in advance.
Failed: Unexpected difference in fileset version 6: 04.10.2022 08:11:19 (database id: 3), found 40235 entries, but expected 40240
Details: Duplicati.Library.Interface.UserInformationException: Unexpected difference in fileset version 6: 04.10.2022 08:11:19 (database id: 3), found 40235 entries, but expected 40240
bei Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
bei Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass34_0.<VerifyConsistencyAsync>b__0()
bei Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.<RunOnMain>b__0()
bei Duplicati.Library.Main.Operation.Common.SingleRunner.<DoRunOnMain>d__2`1.MoveNext()
--- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
--- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
bei Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result)
bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
Log data:
2022-11-21 08:16:16 +01 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Unexpected difference in fileset version 6: 04.10.2022 08:11:19 (database id: 3), found 40235 entries, but expected 40240
bei Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
bei Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass34_0.<VerifyConsistencyAsync>b__0()
bei Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.<RunOnMain>b__0()
bei Duplicati.Library.Main.Operation.Common.SingleRunner.<DoRunOnMain>d__2`1.MoveNext()
--- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
Got a link or a username? Your current user is newly made. If no history, need to start this all over again.
https://github.com/duplicati/duplicati/releases doesn’t show any obvious code change since current Beta.
There’s no lack of open issues (but a distinct lack of volunteers to help with them). Knowing which issue possibly is relevant would need the usual collection of information, ideally with logs and maybe DB info…
Rare problems are always harder to troubleshoot. The good/bad news is your users see it fairly often, but who’s going to be able to supply information, answer questions, etc. on what might be causing the issue?
I am new to this forum. One of our users reported the issue and I accidentally copied too much text from his email. This is my first time reporting this issue here, sorry.
Thanks for clarifying, but what are you and users now willing and able to do to help assist or workaround?
There’s almost no specific information in this post, or even general information, e.g. on OS or destination.
Part of that is due to Duplicati’s too-brief default messages, thus the need for log-file and log-file-log-level. Creating a bug report and posting a link to it can be done after the error, and reduces some need for logs.
What led up to an error can be found best from a log file, and Information is a fairly lightweight log level.
Logging can go far higher, and someone who is willing to help would be very valuable. A reproducible test involving non-sensitive data would be ideal but least likely, so we take what we can get. What can we get?
It would also help to state the relationship between topic title on “files that are missing” and the stack trace which doesn’t show that error, but shows a different one. If you don’t want to pass messages around, any person who is seeing either problem and is willing to look into it can join as a user, and speak on it directly.
I´ve got pretty much the same issue.
I´m running two Jobs, one runs without errors, the other job basically runs once, then the error occurs.
Both jobs have the same Cloud-Storage as a target, with different Target directories of course.
Running the repair doesn´t do anything for us. I´ve deleted the Database for the job, then it runs once again, then it´s done and only throws error… “Found 5291 files that are missing from the remote storage, please run repair”
Weird thing is: it also shows the job as if it never has run in the main overview.
The General log is just empty, nothing to see there. In the remote logs there are some entries, but they in themselves show nothing.
Hi,
I’m very familiar with the initially mentioned error message but I haven’t analyzed the underlying problem.
However, for me adding the following option to the backup job helped: rebuild-missing-dblock-files
yes, there are files on the cloud storage. It´s a Hetzner Storagebox. As far as i can tell, there are 8804 duplicati files in that cloud-folder.
I have created the job again, entirely from scratch, the issue still persists, job runs once, then it errors out.
Log File gives me this:
2022-12-01 04:12:54 +00 - [Warning-Duplicati.Library.Main.Controller-DeprecatedOption]: The option log-level is deprecated: Use the log-file-log-level and console-log-level options instead
2022-12-01 04:12:54 +00 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started
2022-12-01 04:13:18 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()
2022-12-01 04:13:20 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: ()
2022-12-01 04:13:20 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-20221130T115544Z.dlist.zip.aes
2022-12-01 04:13:20 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-b0b3777fa58c9431e898d10f5d378166b.dblock.zip.aes
2022-12-01 04:13:20 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-bd7adc7ff26324833a77f59fc146ce45d.dblock.zip.aes
2022-12-01 04:13:20 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-bf0be90e45548447988dfa965e7fb3094.dblock.zip.aes
2022-12-01 04:13:20 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-bf9ecdf3d9275496cb82ad17a245db191.dblock.zip.aes
... thats going on for a while and all the "missing" files...
2022-12-01 04:13:21 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-icd12d2fa70b94859b1c62a8f58a7e169.dindex.zip.aes
2022-12-01 04:13:21 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i15d2a463e3454aa3aad4f20e449e3208.dindex.zip.aes
2022-12-01 04:13:21 +00 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i62ba2627721a4f26a453cd7e77d6cc7d.dindex.zip.aes
2022-12-01 04:13:21 +00 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteFiles]: Found 5297 files that are missing from the remote storage, please run repair
2022-12-01 04:13:21 +00 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Found 5297 files that are missing from the remote storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x001fd] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0011d] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00452] in <e60bc008dd1b454d861cfacbdd3760b9>:0
root@cloudbackup:~# mono /usr/lib/duplicati/Duplicati.CommandLine.BackendTool.exe list "webdavs://XXXXX.your-storagebox.de:443/\\XXXX-DaSi?auth-username=XXXXXXX&auth-password=XXXXXX"
Name Dir/File LastChange Size
root@cloudbackup:~#
root@cloudbackup:~# mono /usr/lib/duplicati/Duplicati.CommandLine.BackendTester.exe "webdavs://XXXXXXX.your-storagebox.de:443/\\test?auth-username=XXXXXXXXXXXX&auth-password=XXXXXXXXX"
Checking DNS names used by this backend...
XXXXXX.your-storagebox.de
Starting run no 1
Generating file 0 (32.40 MB)
Generating file 1 (14.53 MB)
Generating file 2 (16.42 MB)
Generating file 3 (9.25 MB)
Generating file 4 (4.58 MB)
Generating file 5 (48.92 MB)
Generating file 6 (44.33 MB)
Generating file 7 (24.23 MB)
Generating file 8 (15.19 MB)
Generating file 9 (25.36 MB)
Uploading wrong files ...
Generating file 10 (1.03 KB)
Uploading file 0, 1.03 KB ... done!
Uploading file 0, 1.03 KB ... done!
Uploading file 9, 1.03 KB ... done!
Uploading files ...
Uploading file 0, 32.40 MB ... done!
Uploading file 1, 14.53 MB ... done!
Uploading file 2, 16.42 MB ... done!
Uploading file 3, 9.25 MB ... done!
Uploading file 4, 4.58 MB ... done!
Uploading file 5, 48.92 MB ... done!
Uploading file 6, 44.33 MB ... done!
Uploading file 7, 24.23 MB ... done!
Uploading file 8, 15.19 MB ... done!
Uploading file 9, 25.36 MB ... done!
Verifying file list ...
*** File with name omJeJgrqJg7sOCfKKKypuAkRpSDCLgEQUIGT4cmnKe9ah3B3gjBu6TAWb1XZLX was uploaded but not found afterwards
*** File with name DhzNQFZGM7V3jbsFbQmYZdmtO8RJ3ck4gybikQqg8XhtGnaSosLP was uploaded but not found afterwards
*** File with name gN2UtX1HIiRdBAZJSU6aWCEI7NMvUiqpWNF7uMoLO68lJSggWROjfCgMjtXOkFWz2CqX53ofF was uploaded but not found afterwards
*** File with name SoM8XyRDB1y3D was uploaded but not found afterwards
*** File with name JJwsqDf2O7VnYIskuL6OYdd7 was uploaded but not found afterwards
*** File with name r25e23IjDARC8ZYkoHf was uploaded but not found afterwards
*** File with name rF52eouIza6NCD6IP2iZrbYg71sWD27HFn4gXWoAfdrsp8D8yDRiDRJpPX5n5HeGdxhoJZU was uploaded but not found afterwards
*** File with name acVCcyyFN was uploaded but not found afterwards
*** File with name Sw3zqQW3uA1IibFWb9M1zjrRZxN7F6DqaCHsPONky was uploaded but not found afterwards
*** File with name D2Sg9RYFjki9b81Fk47jqMJIUdPzWNXHk7qlLrbGUPSR was uploaded but not found afterwards
Downloading files
Downloading file 0 ... done
Checking hash ... done
Downloading file 1 ... done
Checking hash ... done
Downloading file 2 ... done
Checking hash ... done
Downloading file 3 ... done
Checking hash ... done
Downloading file 4 ... done
Checking hash ... done
Downloading file 5 ... done
Checking hash ... done
Downloading file 6 ... done
Checking hash ... done
Downloading file 7 ... done
Checking hash ... done
Downloading file 8 ... done
Checking hash ... done
Downloading file 9 ... done
Checking hash ... done
Deleting files...
Checking retrieval of non-existent file...
*** Retrieval of non-existent file failed: System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0
at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0
at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0
at Duplicati.Library.Backend.WEBDAV.Get (System.String remotename, System.IO.Stream stream) [0x00117] in <5df020eccaa84287a3c79d1a4bf95fc9>:0
at Duplicati.Library.Backend.WEBDAV.Get (System.String remotename, System.String filename) [0x00007] in <5df020eccaa84287a3c79d1a4bf95fc9>:0
at Duplicati.CommandLine.BackendTester.Program.Run (System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, System.Boolean first) [0x00940] in <de062425883d470b89e5e32666a73638>:0
*** Retrieval of non-existent file should have failed with FileMissingException
root@cloudbackup:~#
The Backendtest was the second run… before the files got deleted, i snapped this picture of the cloudstorage, so something is going wrong here…
So… i´ve now deleted all files and folders in the cloud storage that were created by the past runnings of the job and created a complete fresh job with ssh connection instead of Webdav. It is still running, but the Duplicati.CommandLine.BackendTool.exe list is giving me a list of files now…
So i guess it should work now.
