Delete of the specific version using GUI Commandline option is what I’ve had to do, however I’ve only seen this at the time when I tried –skip-files-larger-than but received an unexpected mess. I couldn’t reproduce it, but seeing this new topic led me to summarize an Information-level log. Below is a writeup of that summary, condensed into operations and results, with odd results being supplemented by examples from the full log. I’ve been on canary (now 2.0.3.9), set no special metadata options, but do have --snapshot-policy=On
2018-07-19 13:59:46 Backup, normal
2018-07-19 14:19:58 Backup, unusual. Only Put a dblock and dindex
"removing file listed as Temporary" but didn't actually do Delete
2018-07-19 14:39:12 Backup, channel is retired, task was canceled
Also, most of the metadata processing was beyond the backup range
2018-07-19 14:43:42 Backup, Unexpected difference
2018-07-19 14:51:37 Repair
2018-07-19 14:51:56 Backup, channel is retired, UNIQUE constraint
2018-07-19 15:27:28 Backup, Unexpected difference
2018-07-19 15:30:58 Repair, Recreate
2018-07-19 15:33:13 Backup, Metadata was reported, channel is retired, UNIQUE constraint
2018-07-19 15:35:55 Backup, Unexpected difference
2018-07-19 15:49:59 Delete
2018-07-19 15:50:47 Backup, normal
2018-07-19 15:55:37 Backup, channel is retired, task was canceled
2018-07-20 09:57:29 Unexpected difference
Example of "channel is retired", "beyond the backup range", "task was canceled", and "Unexpected difference":
2018-07-19 14:39:55 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\zFTPServer\CBB_HP4\C$\Users\Ted\AppData\Roaming\Mozilla\Firefox\Profiles\3w8afwwg.default\storage\default\https+++www.youtube.com\idb\3211250388sbwdpsunsohintoatciif.sqlite$\20170809230042\3211250388sbwdpsunsohintoatciif.sqlite
CoCoL.RetiredException: The channel is retired
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
2018-07-19 14:39:55 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b0acad27920b44394abc50c52e6e2b450.dblock.zip.aes (124.76 KB)
2018-07-19 14:39:59 -04 - [Information-Duplicati.Library.Modules.Builtin.SendMail-SendMailComplete]: Email sent successfully using server: smtp://smtpauth.earthlink.net:587/?starttls=always
2018-07-19 14:39:59 -04 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Backup has failed with error: A task was canceled.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
2018-07-19 14:42:22 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started
2018-07-19 14:42:22 -04 - [Information-Duplicati.Library.Main.Controller-CompletedOperation]: The operation List has completed
(six more pairs)
2018-07-19 14:43:42 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started
2018-07-19 14:44:21 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Exception: Unexpected difference in fileset 319, found 807 entries, but expected 808
at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
Example of "channel is retired" and "UNIQUE constraint":
2018-07-19 14:52:36 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\Users\Ted\Documents\My Kindle Content\B002B55AEI_EBOK\CR!VC2GAHGBKX24V3AJ5VK3NF1PN9EN.azw.res
CoCoL.RetiredException: The channel is retired
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
2018-07-19 14:52:36 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bc014ba9cccac45028b8d351c472830d9.dblock.zip.aes (1.90 KB)
2018-07-19 14:52:39 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Data.SQLite.SQLiteException (0x80004005): constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
at System.Data.SQLite.SQLiteDataReader.NextResult()
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
at Duplicati.Library.Main.Database.LocalBackupDatabase.AddUnmodifiedFile(Int64 fileid, DateTime lastmodified, IDbTransaction transaction)
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
Example of "Metadata was reported", "channel is retired", and "UNIQUE constraint":
2018-07-19 15:33:57 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-UnexpextedMetadataLookup]: Metadata was reported as not changed, but still requires being added?
Hash: ff5/v9LfE0FV1mvtfS3NPuvO+eg5j5lB47OnboTj/X8=, Length: 137, ID: 88
2018-07-19 15:33:57 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\Users\Ted\Documents\size test\short.txt
CoCoL.RetiredException: The channel is retired
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
...
2018-07-19 15:33:57 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-ProcessingMetadataFailed]: Failed to process entry, path: C:\Users\Ted\Documents\My Kindle Content\B002B55AEI_EBOK\CR!VC2GAHGBKX24V3AJ5VK3NF1PN9EN.azw.res
CoCoL.RetiredException: The channel is retired
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CoCoL.Channel`1.<WriteAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
2018-07-19 15:33:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bb1f7dbf681fc44cd94a27a7ee4d0c988.dblock.zip.aes (1.92 KB)
2018-07-19 15:33:59 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Data.SQLite.SQLiteException (0x80004005): constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
at System.Data.SQLite.SQLiteDataReader.NextResult()
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
at Duplicati.Library.Main.Database.LocalBackupDatabase.AddUnmodifiedFile(Int64 fileid, DateTime lastmodified, IDbTransaction transaction)
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
Across this four month log, I usually see some "Metadata was reported", and a few "UNIQUE constraint".
"channel is retired", "task was canceled", and "Unexpected difference" were only at the troubled time.
I can’t say what an older release would have done, because I’ve only seen the large-scale mess on 2.0.3.9.
I’m hoping some experts can sort out the chain-reaction effects from things that might share a single cause, however out of politeness to this topic, perhaps other topics can get talk that’s not “Unexpected difference”, where I’m contributing some cases plus some context that looks like it might possibly lead into that problem.