I’m willing to help troubleshoot this if anyone wants to send over specific debugging instructions.
These errors only appear with the snapshot-policy enabled, at least for me
Thanks for that - and I apologize if you’ve already said this above, but if you set --snapshot-policy=off do the errors go away for you like they did for @Ferdis?
No, changing the snapshot policy to “off” did not resolve the issue.
* Aug 13, 2018 11:48 AM: Failed while executing "Backup" with id: 3
System.AggregateException: One or more errors occurred. ---> System.AggregateException: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 ---> System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() --- End of inner exception stack trace --- at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() --- End of inner exception stack trace --- at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method) at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter) at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue) ---> (Inner Exception #0) System.AggregateException: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 ---> System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() --- End of inner exception stack trace --- at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() ---> (Inner Exception #0) System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()<--- ---> (Inner Exception #1) System.AggregateException: One or more errors occurred. ---> System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls) at Duplicati.Library.Logging.Log.get_CurrentScope() at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments) at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex) at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext() at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext() --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls) at Duplicati.Library.Logging.Log.get_CurrentScope() at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments) at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex) at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext() at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()<--- <--- <---I have remote backups at box.com and Azure. Both had problems with the errors above. Disabling snapshot (windows 10 pro latest updates) solved the issues.
During update with snapshot on, i have noticed it was processing files i did not checked for backups.
@backupti, @Ferdis, @ts678, @belidzs, @przemas75:
Thanks for reporting.
The error is that Duplicati is attempting to insert the same file twice for some reason.
If one of you who did not downgrade could try to add:
--log-file-log-filter=+*.FileEntry*
--log-file=<path-to-file>
and then run the backup again to collect log data. The last line before the error should be: Skipped checking file, because no metadata was updated <path-to-error-file>
Then check in the log if that file is reported elsewhere.
Yes, that sounds like this issue: Processing files which are not in backup sources
Thinking about it… if the snapshot issue causes unrelated files to be added, it could be that the same file is actually processed twice, which would give this exact exception.
Can anyone confirm if there is some weirdness that causes the same paths to be included multiple times? Like --symlink-policy=follow and then having a cyclic symlink?
Also: Anyone experiencing this on non-Windows?
Snapshot switched on, and the log created. Now, i have:
2018-08-14 11:55:08 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-SkipCheckNoTimestampChange]: Skipped checking file, because timestamp was not updated C:\Users\USER\AppData\Roaming\Mozilla\Firefox\Profiles\qf9gdww9.default\storage\default\https+++www.wp.pl\cache\morgue\193{91a6a7e5-00b9-4567-abb9-0b3720f614c1}.final
2018-08-14 11:55:08 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-AddDirectory]: Adding directory C:\Windows\
First line with date is ok, its included for backup, next line with date (and directory c:\Windows) obviously i dont want to backup
With option --symlink-policy=follow and snapshot on it went ok with some warnings (i’d say usual ones).
Warning-Duplicati.Library.Main.Operation.Backup.UploadSyntheticFilelist-MissingTemporaryFilelist
and
Metadata was reported as not changed, but still requires being added?
Duplicati 2.0.3.9 and windows10x64 (latest updates).
With --symlink-policy=store it fails.
snapshot on
error:
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID ---> System.Data.SQLite.SQLiteException: constraint failed
Duplicati seems to add Windows junctions via AddSymlinkEntryAsync which seems oddly common on stacks:
N backupti
Y belidzs
Y prez
Y ts678
Y jarmo
When I tried catching this exception, I think the first one I caught was Documents. NTFSLinksView shows me:
Application Data AppData\Roaming
Cookies AppData\Local\Microsoft\Windows\INetCookies
Local Settings AppData\Local
My Documents Documents
NetHood AppData\Roaming\Microsoft\Windows\Network Shortcuts
PrintHood AppData\Roaming\Microsoft\Windows\Printer Shortcuts
Recent AppData\Roaming\Microsoft\Windows\Recent
SendTo AppData\Roaming\Microsoft\Windows\SendTo
Start Menu AppData\Roaming\Microsoft\Windows\Start Menu
Templates AppData\Roaming\Microsoft\Windows\Templates
There are certainly some cycles there (depending on processing order) but why isn’t it breaking all the time?
Database viewing at exception time didn’t indicate the issue, but I’m not sure how timing of the activities runs.
I’m using Duplicati 2.0.3.9 and UNIQUE error is appear only sometimes with snapshot policy auto or on. Bug is same as
Processing files which are not in backup sources
Duplicati is scaning some directory and on last file in directory at once starts scan whole disk.
2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\DOPAL\web\stats\webalizer.hist 2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-AddDirectory]: Adding directory C:\DOPAL\web\mail\ 2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-SkipCheckNoTimestampChange]: Skipped checking file, because timestamp was not updated C:\DOPAL\web\stats\webalizer.hist 2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-ExcludingPathFromAttributes]: Excluding path due to attribute filter: C:\$Recycle.Bin\ 2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\$WINDOWS.~BT\
and it scans source directories again and files which scan before.
Version 2.0.3.10 now has fixes for VSS as well as handling the “UNIQUE constraint” issues with a warning instead of a stop.
Great work! After upgrading to version 2.0.3.10 no more UNIQUE errors with snaphot policy turned on. I have already made 3 backups without this error.
1 Like
My issue appears to have been resolved as well!!
1 Like
Thanks for reporting! Very happy to hear that it is now resolved.
I’ve changed the solution from the workaround post to the update one so new readers will know it’s safe to move to 2.0.3.10. 