UNIQUE constraint failed with SIA (Support Request) [2.0.3.6]

Thanks for letting us know - before I take the version number of the topic title, can you confirm that you’re getting this with SIA?

Did you try downgrading to an earlier version (maybe all the way back to 2.0.3.5) to see if the error goes away?

Sorry for the confusion, it seems like my issue is not exactly same as I’m using SFTP and not SIA as backend.

However, I ended up exporting the jobs, deleting the local database, downgrading to 2.0.3.3, importing the jobs back then recreated the databases.

This is my stack trace which made me suspect I have the same issue:

    Failed: One or more errors occurred.
    Details: System.AggregateException: One or more errors occurred. ---> System.AggregateException: constraint failed
    UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID ---> System.Data.SQLite.SQLiteException: constraint failed
    UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
    at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
    at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
    at System.Data.SQLite.SQLiteDataReader.NextResult()
    at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
    at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
    at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
    at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
    at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
    --- End of inner exception stack trace ---
    at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
    --- End of inner exception stack trace ---
    at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
    at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
    at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
    ---> (Inner Exception #0) System.AggregateException: constraint failed
    UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID ---> System.Data.SQLite.SQLiteException: constraint failed
    UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
    at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
    at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
    at System.Data.SQLite.SQLiteDataReader.NextResult()
    at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
    at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
    at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
    at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
    at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
    --- End of inner exception stack trace ---
    at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
    ---> (Inner Exception #0) System.Data.SQLite.SQLiteException (0x80004005): constraint failed
    UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
    at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
    at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
    at System.Data.SQLite.SQLiteDataReader.NextResult()
    at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
    at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
    at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
    at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
    at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()<---

    ---> (Inner Exception #1) System.AggregateException: One or more errors occurred. ---> System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
    at Duplicati.Library.Logging.Log.get_CurrentScope()
    at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments)
    at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex)
    at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext()
    at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
    at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext()
    at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
    --- End of inner exception stack trace ---
    ---> (Inner Exception #0) System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
    at Duplicati.Library.Logging.Log.get_CurrentScope()
    at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments)
    at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex)
    at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext()
    at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
    at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext()
    at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()<---
    <---
    <---


    Log data:
    2018-07-17 03:00:40 +02 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
    System.Data.SQLite.SQLiteException (0x80004005): constraint failed
    UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
    at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
    at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
    at System.Data.SQLite.SQLiteDataReader.NextResult()
    at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
    at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
    at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
    at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
    at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
    at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()

No problem - I was just trying to narrow down potential commonalities to the issue.

Since your error is the same as the original poster but you’re using a different destination that points to the database code (rather than Sia code) in Duplicati as a likely source of the issue.

While not a great “fix” experience (but better than nothing) I’m curious if the export + delete + downgrade + import + recreate solved the problem for you…

Yes, it is working since then. Obviously I’m too afraid to upgrade anytime soon, because these operations took a lot of time and attentions to finish.

Understandable - if you’re in a stable situation, definitely stay there and let us crazier people run with the latest canary versions until the kinks are ironed out. :crazy_face:

This post seems to be collecting these, so I’ll try to summarize what I’m seeing.
This possibly happens on 2.0.3.6 and above on various databases, in several ways.
To show the several ways, some simplified stacks show commonality and difference.
Note we have two flavors of add, and two flavors of preprocessing driving them:

@backupti on 2.0.3.6 with SIA got stack
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
LocalBackupDatabase.AddUnmodifiedFile
Backup.FilePreFilterProcess

The fix there was 2.0.3.5, which is the one before concurrenct processing came in.
This has the feeling of a concurrency issue, with things tripping over each other.
My attempt to follow the code didn’t get far though, so better help may be needed.

@belidzs on 2.0.3.9 with SFTP got stack
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
LocalBackupDatabase.AddFile
Backup.MetadataPreProcess.

@prez on 2.0.3.9 on #4007 got stack

UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
LocalBackupDatabase.AddFile
Backup.MetadataPreProcess.

But because that was directed here, let me announce

@ts678 on 2.0.3.9 with B2 got stack
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
LocalBackupDatabase.AddFile
Backup.MetadataPreProcess.

and like the one from prez, this used to work fine.
Unlike prez and some others, I didn’t need repairs.
Possibly I scared something away with the debugger.

For my history, my server log goes back about 4 months. Only previous was on 7/19:
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
LocalBackupDatabase.AddUnmodifiedFile
Backup.FilePreFilterProcess

From my small daily can’t-think-of-anything-I-changed-lately-in-configuration backup:

Failed: One or more errors occurred.
Details: System.AggregateException: One or more errors occurred. ---> System.AggregateException: constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID ---> System.Data.SQLite.SQLiteException: constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
---> (Inner Exception #0) System.AggregateException: constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID ---> System.Data.SQLite.SQLiteException: constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
---> (Inner Exception #0) System.Data.SQLite.SQLiteException (0x80004005): constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()<---

---> (Inner Exception #1) System.AggregateException: One or more errors occurred. ---> System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
   at Duplicati.Library.Logging.Log.get_CurrentScope()
   at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments)
   at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex)
   at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext()
   at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
   at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext()
   at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
   at Duplicati.Library.Logging.Log.get_CurrentScope()
   at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments)
   at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c.<Run>b__1_3(String rootpath, String errorpath, Exception ex)
   at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext()
   at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
   at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<ExpandWorkList>d__4.MoveNext()
   at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()<---
<---
<---


Log data:
2018-07-26 13:01:04 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Data.SQLite.SQLiteException (0x80004005): constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass9_0.<AddSymlinkEntryAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddSymlinkToOutputAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()

I’m pretty sure something’s going wrong with SQLite usage. I know logging is on its way that might help.

Adding cross references.

jarmo in GitHub issue #3315 got this on 2.0.3.9 seemingly right after upgrading 2.0.3.5.

ts678 updated #4286 with two examples of “Unique constraint” (and lots of other errors).
There seem to be some patterns in the errors there, but expert interpretation is needed.
If anyone else has found “UNIQUE constraint” in a string of errors, the details might help.

I’m willing to help troubleshoot this if anyone wants to send over specific debugging instructions.

These errors only appear with the snapshot-policy enabled, at least for me

Thanks for that - and I apologize if you’ve already said this above, but if you set --snapshot-policy=off do the errors go away for you like they did for @Ferdis?

No, changing the snapshot policy to “off” did not resolve the issue.

* Aug 13, 2018 11:48 AM: Failed while executing "Backup" with id: 3

System.AggregateException: One or more errors occurred. ---&gt; System.AggregateException: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 ---&gt; System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.&lt;&gt;c__DisplayClass31_0.&lt;VerifyConsistencyAsync&gt;b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.&lt;&gt;c__DisplayClass6_0.&lt;RunOnMain&gt;b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.&lt;&gt;c__DisplayClass5_0`1.&lt;&lt;DoRunOnMain&gt;b__1&gt;d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.&lt;RunAsync&gt;d__19.MoveNext() --- End of inner exception stack trace --- at Duplicati.Library.Main.Operation.BackupHandler.&lt;RunAsync&gt;d__19.MoveNext() --- End of inner exception stack trace --- at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.&lt;&gt;c__DisplayClass13_0.&lt;Backup&gt;b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]&amp; paths, IFilter&amp; filter, Action`1 method) at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter) at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue) ---&gt; (Inner Exception #0) System.AggregateException: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 ---&gt; System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.&lt;&gt;c__DisplayClass31_0.&lt;VerifyConsistencyAsync&gt;b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.&lt;&gt;c__DisplayClass6_0.&lt;RunOnMain&gt;b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.&lt;&gt;c__DisplayClass5_0`1.&lt;&lt;DoRunOnMain&gt;b__1&gt;d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.&lt;RunAsync&gt;d__19.MoveNext() --- End of inner exception stack trace --- at Duplicati.Library.Main.Operation.BackupHandler.&lt;RunAsync&gt;d__19.MoveNext() ---&gt; (Inner Exception #0) System.Exception: Unexpected difference in fileset 22, found 1343 entries, but expected 1344 at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) at Duplicati.Library.Main.Operation.Backup.BackupDatabase.&lt;&gt;c__DisplayClass31_0.&lt;VerifyConsistencyAsync&gt;b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.&lt;&gt;c__DisplayClass6_0.&lt;RunOnMain&gt;b__0() at Duplicati.Library.Main.Operation.Common.SingleRunner.&lt;&gt;c__DisplayClass5_0`1.&lt;&lt;DoRunOnMain&gt;b__1&gt;d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.&lt;RunAsync&gt;d__19.MoveNext()&lt;--- ---&gt; (Inner Exception #1) System.AggregateException: One or more errors occurred. ---&gt; System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls) at Duplicati.Library.Logging.Log.get_CurrentScope() at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments) at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.&lt;&gt;c.&lt;Run&gt;b__1_3(String rootpath, String errorpath, Exception ex) at Duplicati.Library.Utility.Utility.&lt;EnumerateFileSystemEntries&gt;d__23.MoveNext() at System.Linq.Enumerable.&lt;SelectManyIterator&gt;d__17`2.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.&lt;ExpandWorkList&gt;d__4.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.&lt;&gt;c__DisplayClass1_0.&lt;&lt;Run&gt;b__0&gt;d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CoCoL.AutomationExtensions.&lt;RunTask&gt;d__10`1.MoveNext() --- End of inner exception stack trace --- ---&gt; (Inner Exception #0) System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls) at Duplicati.Library.Logging.Log.get_CurrentScope() at Duplicati.Library.Logging.Log.WriteMessage(LogMessageType type, String tag, String id, Exception ex, String message, Object[] arguments) at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.&lt;&gt;c.&lt;Run&gt;b__1_3(String rootpath, String errorpath, Exception ex) at Duplicati.Library.Utility.Utility.&lt;EnumerateFileSystemEntries&gt;d__23.MoveNext() at System.Linq.Enumerable.&lt;SelectManyIterator&gt;d__17`2.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.&lt;ExpandWorkList&gt;d__4.MoveNext() at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.&lt;&gt;c__DisplayClass1_0.&lt;&lt;Run&gt;b__0&gt;d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CoCoL.AutomationExtensions.&lt;RunTask&gt;d__10`1.MoveNext()&lt;--- &lt;--- &lt;---

I have remote backups at box.com and Azure. Both had problems with the errors above. Disabling snapshot (windows 10 pro latest updates) solved the issues.
During update with snapshot on, i have noticed it was processing files i did not checked for backups.

@backupti, @Ferdis, @ts678, @belidzs, @przemas75:
Thanks for reporting.
The error is that Duplicati is attempting to insert the same file twice for some reason.

If one of you who did not downgrade could try to add:

--log-file-log-filter=+*.FileEntry*
--log-file=<path-to-file>

and then run the backup again to collect log data. The last line before the error should be: Skipped checking file, because no metadata was updated <path-to-error-file>

Then check in the log if that file is reported elsewhere.

Yes, that sounds like this issue: Processing files which are not in backup sources

Thinking about it… if the snapshot issue causes unrelated files to be added, it could be that the same file is actually processed twice, which would give this exact exception.

Can anyone confirm if there is some weirdness that causes the same paths to be included multiple times? Like --symlink-policy=follow and then having a cyclic symlink?

Also: Anyone experiencing this on non-Windows?

Snapshot switched on, and the log created. Now, i have:

2018-08-14 11:55:08 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-SkipCheckNoTimestampChange]: Skipped checking file, because timestamp was not updated C:\Users\USER\AppData\Roaming\Mozilla\Firefox\Profiles\qf9gdww9.default\storage\default\https+++www.wp.pl\cache\morgue\193{91a6a7e5-00b9-4567-abb9-0b3720f614c1}.final
2018-08-14 11:55:08 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-AddDirectory]: Adding directory C:\Windows\

First line with date is ok, its included for backup, next line with date (and directory c:\Windows) obviously i dont want to backup

With option --symlink-policy=follow and snapshot on it went ok with some warnings (i’d say usual ones).
Warning-Duplicati.Library.Main.Operation.Backup.UploadSyntheticFilelist-MissingTemporaryFilelist
and
Metadata was reported as not changed, but still requires being added?

Duplicati 2.0.3.9 and windows10x64 (latest updates).

With --symlink-policy=store it fails.
snapshot on
error:
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID ---> System.Data.SQLite.SQLiteException: constraint failed

Duplicati seems to add Windows junctions via AddSymlinkEntryAsync which seems oddly common on stacks:

N backupti
Y belidzs
Y prez
Y ts678
Y jarmo

When I tried catching this exception, I think the first one I caught was Documents. NTFSLinksView shows me:

Application Data	AppData\Roaming
Cookies			AppData\Local\Microsoft\Windows\INetCookies
Local Settings		AppData\Local
My Documents		Documents
NetHood			AppData\Roaming\Microsoft\Windows\Network Shortcuts
PrintHood		AppData\Roaming\Microsoft\Windows\Printer Shortcuts
Recent			AppData\Roaming\Microsoft\Windows\Recent
SendTo			AppData\Roaming\Microsoft\Windows\SendTo
Start Menu		AppData\Roaming\Microsoft\Windows\Start Menu
Templates		AppData\Roaming\Microsoft\Windows\Templates

There are certainly some cycles there (depending on processing order) but why isn’t it breaking all the time?

Database viewing at exception time didn’t indicate the issue, but I’m not sure how timing of the activities runs.

I’m using Duplicati 2.0.3.9 and UNIQUE error is appear only sometimes with snapshot policy auto or on. Bug is same as
Processing files which are not in backup sources

Duplicati is scaning some directory and on last file in directory at once starts scan whole disk.

2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\DOPAL\web\stats\webalizer.hist
2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.FileEntry-AddDirectory]: Adding directory C:\DOPAL\web\mail\
2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-SkipCheckNoTimestampChange]: Skipped checking file, because timestamp was not updated C:\DOPAL\web\stats\webalizer.hist
2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-ExcludingPathFromAttributes]: Excluding path due to attribute filter: C:\$Recycle.Bin\
2018-08-17 17:14:23 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\$WINDOWS.~BT\

and it scans source directories again and files which scan before.

Version 2.0.3.10 now has fixes for VSS as well as handling the “UNIQUE constraint” issues with a warning instead of a stop.