"Unexpected difference in fileset" errors [2.0.3.9]


#1

In my testing of various scenarios in 2.0.3.9 canary I’ve managed to somehow / accidentally recreate the “Unexpected difference in fileset” error that some other users have run into. (Detailed error below.)

For most of them, a database recreate seemed to resolve the issue but I believe I have a secondary issue (missing file?) that is not allowing that resolution to work for me.

In the end, I believe I had to:

  1. do a database Repair (just to make sure DB was in good shape)
  2. delete all dlist files (because I couldn’t easily figure out which one belonged to fileset / backup #54)
  3. do a database Recreate (the lack of dlist files forced full downloads of all dblock files, but bandwidth isn’t an issue for me)
  4. do another database Repair

After all that, I had ONE backup run work correctly after which I noticed the “Restore from” list showed jobs NEWER than the job menu’s “Last successful backup” date, and further backups failed with “Unexpected number of remote volumes marked as deleted” global errors.

Those “…marked as deleted…” errors go away if I change from a custom --retention-policy to a “Backup retention” of “Keep all backups”. Putting the custom retention policy back in brings the “Unexpected number of remote volumes marked as deleted” error back, even after successful “Keep all backups” runs.

Oddly, the error even happens with a custom retention policy of U:U. :frowning:

Some other things I tried that did NOT resolve the original issue for me include:
  • A database Repair which finished with no errors
  • A database Recreate which finished with a “missing files” error
  • Rolling back to 2.0.3.7 (the oldest I could get to due to database version 8 vs 7 issues) - note that rolling back to 2.0.3.5 seemed to work for somebody
  • Deleting all the dindex files (in case they had an orphaned reference to a deleted file) followed by a database Rebuild, but it did not resolve the original issue. A subsequent database Recreate still reported the missing file and also did not resolve the original issue.
  • At one point @kenkendk suggested to another user to delete the troublesome version. In my case that’s version #54 but while the main job page says I have 54 versions, the (zero based) Restore page only shows 53 so I’m not sure if I’m supposed to be deleting version 53 or if there’s a phantom version 54 floating around somewhere.

The error:

System.AggregateException: One or more errors occurred.
---> System.AggregateException: Unexpected difference in fileset 54, found 13837 entries, but expected 13841
---> System.Exception: Unexpected difference in fileset 54, found 13837 entries, but expected 13841
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x002bc] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () [0x00038] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <a6f810169ec746eeb2789b925e2858c0>:0
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x003c9] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x0102c] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of inner exception stack trace ---
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x0005d] in <6973ce2780de4b28aaa2c5ffc59993b1>:0
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x00008] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Controller+<>c__DisplayClass13_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x00035] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0022e] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00068] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00335] in <8486dd191d20467cbf8d627f56ca2e90>:0
---> (Inner Exception #0) System.AggregateException: Unexpected difference in fileset 54, found 13837 entries, but expected 13841
---> System.Exception: Unexpected difference in fileset 54, found 13837 entries, but expected 13841
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x002bc] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () [0x00038] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <a6f810169ec746eeb2789b925e2858c0>:0
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x003c9] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x0102c] in <0ce58d578b8642d49036dc15fbad38f1>:0
---> (Inner Exception #0) System.Exception: Unexpected difference in fileset 54, found 13837 entries, but expected 13841
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x002bc] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () [0x00038] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <a6f810169ec746eeb2789b925e2858c0>:0
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x003c9] in <0ce58d578b8642d49036dc15fbad38f1>:0 <---
---> (Inner Exception #1) System.AggregateException: One or more errors occurred.
---> System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
  at Duplicati.Library.Logging.Log.get_CurrentScope () [0x0004d] in <a0d17c5f1b8942efac55f621de8cc00b>:0
  at Duplicati.Library.Logging.Log.WriteMessage (Duplicati.Library.Logging.LogMessageType type, System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x0001e] in <a0d17c5f1b8942efac55f621de8cc00b>:0
  at Duplicati.Library.Logging.Log.WriteWarningMessage (System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00000] in <a0d17c5f1b8942efac55f621de8cc00b>:0
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c.<Run>b__1_3 (System.String rootpath, System.String errorpath, System.Exception ex) [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Utility.Utility+<EnumerateFileSystemEntries>d__23.MoveNext () [0x001d9] in <5336a8b903594fa5ae5a9692dae4b7fe>:0
  at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x0006f] in <abeb153532fb46718501a600866a8d76>:0
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<ExpandWorkList>d__4.MoveNext () [0x000da] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c__DisplayClass1_0+<<Run>b__0>d.MoveNext () [0x00263] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <a6f810169ec746eeb2789b925e2858c0>:0
  at CoCoL.AutomationExtensions+<RunTask>d__10`1[T].MoveNext () [0x000cc] in <6973ce2780de4b28aaa2c5ffc59993b1>:0
 --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
  at Duplicati.Library.Logging.Log.get_CurrentScope () [0x0004d] in <a0d17c5f1b8942efac55f621de8cc00b>:0
  at Duplicati.Library.Logging.Log.WriteMessage (Duplicati.Library.Logging.LogMessageType type, System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x0001e] in <a0d17c5f1b8942efac55f621de8cc00b>:0
  at Duplicati.Library.Logging.Log.WriteWarningMessage (System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00000] in <a0d17c5f1b8942efac55f621de8cc00b>:0
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c.<Run>b__1_3 (System.String rootpath, System.String errorpath, System.Exception ex) [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Utility.Utility+<EnumerateFileSystemEntries>d__23.MoveNext () [0x001d9] in <5336a8b903594fa5ae5a9692dae4b7fe>:0
  at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x0006f] in <abeb153532fb46718501a600866a8d76>:0
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<ExpandWorkList>d__4.MoveNext () [0x000da] in <0ce58d578b8642d49036dc15fbad38f1>:0
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c__DisplayClass1_0+<<Run>b__0>d.MoveNext () [0x00263] in <0ce58d578b8642d49036dc15fbad38f1>:0
 --- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <a6f810169ec746eeb2789b925e2858c0>:0
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <a6f810169ec746eeb2789b925e2858c0>:0
  at CoCoL.AutomationExtensions+<RunTask>d__10`1[T].MoveNext () [0x000cc] in <6973ce2780de4b28aaa2c5ffc59993b1>:0
<--- <--- <---

Backup failing due to unexpected difference in fileset
"Unexpected number of remote volumes marked as deleted" error
#2

Delete of the specific version using GUI Commandline option is what I’ve had to do, however I’ve only seen this at the time when I tried –skip-files-larger-than but received an unexpected mess. I couldn’t reproduce it, but seeing this new topic led me to summarize an Information-level log. Below is a writeup of that summary, condensed into operations and results, with odd results being supplemented by examples from the full log. I’ve been on canary (now 2.0.3.9), set no special metadata options, but do have --snapshot-policy=On

2018-07-19 13:59:46 Backup, normal

2018-07-19 14:19:58 Backup, unusual. Only Put a dblock and dindex
"removing file listed as Temporary" but didn't actually do Delete

2018-07-19 14:39:12 Backup, channel is retired, task was canceled
Also, most of the metadata processing was beyond the backup range

2018-07-19 14:43:42 Backup, Unexpected difference

2018-07-19 14:51:37 Repair

2018-07-19 14:51:56 Backup, channel is retired, UNIQUE constraint

2018-07-19 15:27:28 Backup, Unexpected difference

2018-07-19 15:30:58 Repair, Recreate

2018-07-19 15:33:13 Backup, Metadata was reported, channel is retired, UNIQUE constraint

2018-07-19 15:35:55 Backup, Unexpected difference

2018-07-19 15:49:59 Delete

2018-07-19 15:50:47 Backup, normal

2018-07-19 15:55:37 Backup, channel is retired, task was canceled

2018-07-20 09:57:29 Unexpected difference

Example of "channel is retired", "beyond the backup range", "task was canceled", and "Unexpected difference":

2018-07-19 14:39:55 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\zFTPServer\CBB_HP4\C$\Users\Ted\AppData\Roaming\Mozilla\Firefox\Profiles\3w8afwwg.default\storage\default\https+++www.youtube.com\idb\3211250388sbwdpsunsohintoatciif.sqlite$\20170809230042\3211250388sbwdpsunsohintoatciif.sqlite
CoCoL.RetiredException: The channel is retired
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
2018-07-19 14:39:55 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b0acad27920b44394abc50c52e6e2b450.dblock.zip.aes (124.76 KB)
2018-07-19 14:39:59 -04 - [Information-Duplicati.Library.Modules.Builtin.SendMail-SendMailComplete]: Email sent successfully using server: smtp://smtpauth.earthlink.net:587/?starttls=always
2018-07-19 14:39:59 -04 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Backup has failed with error: A task was canceled.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
2018-07-19 14:42:22 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started
2018-07-19 14:42:22 -04 - [Information-Duplicati.Library.Main.Controller-CompletedOperation]: The operation List has completed
(six more pairs)
2018-07-19 14:43:42 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started
2018-07-19 14:44:21 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Exception: Unexpected difference in fileset 319, found 807 entries, but expected 808
   at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()

Example of "channel is retired" and "UNIQUE constraint":

2018-07-19 14:52:36 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\Users\Ted\Documents\My Kindle Content\B002B55AEI_EBOK\CR!VC2GAHGBKX24V3AJ5VK3NF1PN9EN.azw.res
CoCoL.RetiredException: The channel is retired
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
2018-07-19 14:52:36 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bc014ba9cccac45028b8d351c472830d9.dblock.zip.aes (1.90 KB)
2018-07-19 14:52:39 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Data.SQLite.SQLiteException (0x80004005): constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddUnmodifiedFile(Int64 fileid, DateTime lastmodified, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

Example of "Metadata was reported", "channel is retired", and "UNIQUE constraint":

2018-07-19 15:33:57 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-UnexpextedMetadataLookup]: Metadata was reported as not changed, but still requires being added?
Hash: ff5/v9LfE0FV1mvtfS3NPuvO+eg5j5lB47OnboTj/X8=, Length: 137, ID: 88
2018-07-19 15:33:57 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\Users\Ted\Documents\size test\short.txt
CoCoL.RetiredException: The channel is retired
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
...
2018-07-19 15:33:57 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\Users\Ted\Documents\My Kindle Content\B002B55AEI_EBOK\CR!VC2GAHGBKX24V3AJ5VK3NF1PN9EN.azw.res
CoCoL.RetiredException: The channel is retired
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
2018-07-19 15:33:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bb1f7dbf681fc44cd94a27a7ee4d0c988.dblock.zip.aes (1.92 KB)
2018-07-19 15:33:59 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Data.SQLite.SQLiteException (0x80004005): constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddUnmodifiedFile(Int64 fileid, DateTime lastmodified, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

Across this four month log, I usually see some "Metadata was reported", and a few "UNIQUE constraint".
"channel is retired", "task was canceled", and "Unexpected difference" were only at the troubled time.

I can’t say what an older release would have done, because I’ve only seen the large-scale mess on 2.0.3.9.

I’m hoping some experts can sort out the chain-reaction effects from things that might share a single cause, however out of politeness to this topic, perhaps other topics can get talk that’s not “Unexpected difference”, where I’m contributing some cases plus some context that looks like it might possibly lead into that problem.


UNIQUE constraint failed with SIA (Support Request) [2.0.3.6]
#3

I was going to do that but when I went to look at the contents of the version listed in the error message (54) I got an error that it was an invalid version number. There was a version 53 that looked like it either was or was almost the initial backup, so I’m thinking maybe 54 got incompletely cleaned up during retention policy compacting.

Of course it doesn’t help that my current versions 31 and higher were done with 7zip, so that may be the underlying issue in my case.


#4

Hello, I had this problem too and I hope this solution will help somebody:

Duplicati: 2.0.3.11_canary_2018-09-05
Error message probably after crash during backup:

Unexpected difference in fileset 387, found 28618 entries, but expected 28646

I tried:

  • Repair - nothing
  • Verify - nothing
  • Backup - errror message
  • Delete --version=387 - nothing
  • List - I had only 90 versions so I could not delete version 387
  • list-broken-files - some files were found
  • purge-broken-files -some files were deleted

Then I backed up my DB and start recreate. After recreate I can back up without error message.


#5

Thanks for the suggestion, I’ll give it a try. I’m pretty sure I tried it before to no effect, but will message had actually progressed (or been covered up) by Unexpected number of remote volumes marked as deleted. now so hopefully a rebuild will do the trick this time.

Out of curiosity, did you also is 7z at some point in the job history? Hopefully not as that would mean the issue isn’t (just) 7z related.

Edit: Unfortunately it didn’t work. I think I may have somehow resolved the issue (perhaps as you did) but then ran into the ““Unexpected number of remote volumes marked as deleted” error” errors. :frowning:


#6

I might have figured out why @JonMikelV and @mr-flibble had trouble getting the fileset number from their error message to match anything in the regular UI. See Backup failing due to unexpected difference in fileset.

By any chance do either of you have your broken database around? It might be useful to experiment with, but we’d probably want to keep testing from harming anything (including destination data) that’s now running well.


#7

Mine is still erroring out every day. It’s a secondary backup so I figured I’d let it sit until somebody had an idea. :slight_smile:

What do you need from me?


#8

See what you think of this idea, suggest any changes, prepare as needed, try it, and tell everyone if it works. Although this backup doesn’t sound too important, you could try to preserve it in case a next-idea is needed. Maybe it could even be used someday to try new Duplicati code to fix this so such manual work isn’t needed.
EDIT: or is it too late, i.e. has the database with “Unexpected difference in fileset” moved to a different error?

Thanks!


#9

Shoot, you’re right - it has moved on to the unexpected number of remote volumes marked as deleted error. :frowning:

I may have backed up the version that failed with the fileset error. I’ll try to check, but it may take me a while to get to it (just had a drive report read errors on my array).