UNIQUE constraint failed with SIA (Support Request) [2.0.3.6]


#22

I’m willing to help troubleshoot this if anyone wants to send over specific debugging instructions.


#23

These errors only appear with the snapshot-policy enabled, at least for me


#24

Thanks for that - and I apologize if you’ve already said this above, but if you set --snapshot-policy=off do the errors go away for you like they did for @Ferdis?


#25

No, changing the snapshot policy to “off” did not resolve the issue.

* Aug 13, 2018 11:48 AM: Failed while executing "Backup" with id: 3

System.AggregateException: One or more errors occurred. ---> System.AggregateException: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 ---> System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() --- End of inner exception stack trace --- at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() --- End of inner exception stack trace --- at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method) at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter) at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue) ---> (Inner Exception #0) System.AggregateException: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 ---> System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() --- End of inner exception stack trace --- at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() ---> (Inner Exception #0) System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()<--- ---> (Inner Exception #1) System.AggregateException: One or more errors occurred. ---> System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls) at Duplicati.Library.Logging.Log.get_CurrentScope() at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments) at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex) at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext() at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext() --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls) at Duplicati.Library.Logging.Log.get_CurrentScope() at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments) at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex) at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext() at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()<--- <--- <---

#26

I have remote backups at box.com and Azure. Both had problems with the errors above. Disabling snapshot (windows 10 pro latest updates) solved the issues.
During update with snapshot on, i have noticed it was processing files i did not checked for backups.


#27

@backupti, @Ferdis, @ts678, @belidzs, @przemas75:
Thanks for reporting.
The error is that Duplicati is attempting to insert the same file twice for some reason.

If one of you who did not downgrade could try to add:

--log-file-log-filter=+*.FileEntry*
--log-file=<path-to-file>

and then run the backup again to collect log data. The last line before the error should be: Skipped checking file, because no metadata was updated <path-to-error-file>

Then check in the log if that file is reported elsewhere.

Yes, that sounds like this issue: Processing files which are not in backup sources


#28

Thinking about it… if the snapshot issue causes unrelated files to be added, it could be that the same file is actually processed twice, which would give this exact exception.

Can anyone confirm if there is some weirdness that causes the same paths to be included multiple times? Like --symlink-policy=follow and then having a cyclic symlink?

Also: Anyone experiencing this on non-Windows?


#29

Snapshot switched on, and the log created. Now, i have:

2018-08-14 11:55:08 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-SkipCheckNoTimestampChange]: Skipped checking file, because timestamp was not updated C:\Users\USER\AppData\Roaming\Mozilla\Firefox\Profiles\qf9gdww9.default\storage\default\https+++www.wp.pl\cache\morgue\193{91a6a7e5-00b9-4567-abb9-0b3720f614c1}.final
2018-08-14 11:55:08 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-AddDirectory]: Adding directory C:\Windows\

First line with date is ok, its included for backup, next line with date (and directory c:\Windows) obviously i dont want to backup


#30

With option --symlink-policy=follow and snapshot on it went ok with some warnings (i’d say usual ones).
Warning-Duplicati.Library.Main.Operation.Backup.UploadSyntheticFilelist-MissingTemporaryFilelist
and
Metadata was reported as not changed, but still requires being added?

Duplicati 2.0.3.9 and windows10x64 (latest updates).


#31

With --symlink-policy=store it fails.
snapshot on
error:
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID ---> System.Data.SQLite.SQLiteException: constraint failed


#32

Duplicati seems to add Windows junctions via AddSymlinkEntryAsync which seems oddly common on stacks:

N backupti
Y belidzs
Y prez
Y ts678
Y jarmo

When I tried catching this exception, I think the first one I caught was Documents. NTFSLinksView shows me:

Application Data	AppData\Roaming
Cookies			AppData\Local\Microsoft\Windows\INetCookies
Local Settings		AppData\Local
My Documents		Documents
NetHood			AppData\Roaming\Microsoft\Windows\Network Shortcuts
PrintHood		AppData\Roaming\Microsoft\Windows\Printer Shortcuts
Recent			AppData\Roaming\Microsoft\Windows\Recent
SendTo			AppData\Roaming\Microsoft\Windows\SendTo
Start Menu		AppData\Roaming\Microsoft\Windows\Start Menu
Templates		AppData\Roaming\Microsoft\Windows\Templates

There are certainly some cycles there (depending on processing order) but why isn’t it breaking all the time?

Database viewing at exception time didn’t indicate the issue, but I’m not sure how timing of the activities runs.