Constraint failed UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID

Support
1

Getting error message “constraint failed UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID” and backup not proceeding.

Running - Duplicati - 2.0.5.107_canary_2020-05-26 — from previous threads seems like this error appeared a few times but largely around version changes but I have been running current a while and backup has been stable for ages as well.

the “about / show log” about below but not sure how to interpret. Anyone can give insights on how to solve? I would rather not reupload the entire backup set…

code = Constraint (19), message = System.Data.SQLite.SQLiteException (0x800027AF): constraint failed
UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery(IDbCommand self, Boolean writeLog, String cmd, Object[] values)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(Int64 pathprefixid, String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
   at Duplicati.Library.Main.Database.LocalBackupDatabase.AddFile(String filename, DateTime lastmodified, Int64 blocksetID, Int64 metadataID, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass8_0.<AddDirectoryEntryAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<DoRunOnMain>d__2`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<AddFolderToOutputAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<ProcessMetadata>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.Main.Operation.Backup.MetadataPreProcess.<>c__DisplayClass2_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunMainOperation>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)
2

Interpreting the end result is technical, and would be better done by a Duplicati and SQL expert, but it’s still easier than explaining how it happened. The poor end result is probably an attempt to add two files with the same numerical ID, which is obtained from an INSERT into a different table which should give different IDs for every file. That may fail under certain threading conditions, but possibly there are safeguards against it.

If you still have your database sort of close to where it stopped, it might be good to manually save a copy, and also download a database bug report which could maybe be posted later, if it seems like it might help.

Having preserved some history, there’s a chance Duplicati will just pick up and continue with next backup, however sometimes self-check on the database will fail. If you want to test first, hit Verify files button.

If self-check fails with “Unexpected difference in fileset”, good case is if it points to recent version (0 or 1) which delete in Commandline can remove by specifying the version on the --version option. But first let’s save some state if it still exists, and try Verify files. If that works OK, then maybe try to do the backup.

What OS is this? Windows has some weird issues with trailing dots on filenames that sometimes do this. What would be preferable would be for it to be a one-time-thing for you. Even better, the cause gets found.

3

Solved it.

OS is Windows and there was a duplicate subolder entry with a trailing space (so two folders named “other” and "other ")

Removal via command line fixed things; afterwards duplicati could resume easily.

4

I’m glad you found it. The trailing-dot issue I began looking at now has a semi-reproducible failure, and the trailing-space failure might be similar (failure mechanism so far not known). In both cases, folders appear distinct in their parent folder listing, but are sometimes treated as synonymous (sort of like hard links), e.g. Explorer has file placed into non-trailing-space version visible in the with-trailing-space version of folder…

Originally I was using a Command Prompt redirect and \\?\C:\ workaround (needed for trailing dot test), however in the trailing-space test I can just make a file in Explorer once, and have it show up in two spots:

C:\backup source\space_test>tree /f
Folder PATH listing
Volume serial number is E471-3EE4
C:.
├───sub
│       sub.txt
│       sub2.txt
│
└───sub
        sub.txt
        sub2.txt
5

yeh I had similar where a lot of tools, not just duplicati, would get confused and despite the "other " folder being empty; it would mistakenly reuse the non-space “other” folder.

6

Exclude invalid Windows paths #4251 is a much-appreciated developer volunteer’s initial work on this.

1 Like