unique-constraint-error: SOLVED for Windows

Exact error message of a failed backup session using Duplicati:
constraint failed UNIQUE constraint failed: FilesetEntry.FilesetID, FilesetEntry.FileID

This can happen even when Duplicati ran without problems several weeks. Chances are high that this is caused by a cookie file - as it was in my case.

Reason:
Folders in Windows ending with a blank or a dot are problematic and should be avoided. More details can be found here:

Fix:
How to find the problematic directories?
I used Everything of www.voidtools.com
There I entered
folder: regex:".*\.$" OR regex:".* $"
which is looking for folders with an ending blank or dot.
Then just delete or rename them using a bash command line when you are running Windows. The Windows Explorer is not able to help you here.

1 Like

Welcome to the forum @Alex-at-gthb

was a problem I attempted to solve using Duplicati logs and a good tool to search them.
Thanks for an integrated solution. I linked one person with the issue to this topic already.

Thanks for linking!

Actually the situation was like that: underneath my Firefox profile (/cygdrive/c/Users//AppData/Roaming/Mozilla/Firefox/Profiles/.default-release/storage/default) there have been two folders
https+++www.nzz.ch
https+++www.nzz.ch.
easy visible with bash under cygwin. Windows Explorer only saw
https+++www.nzz.ch
because he is not able to differentiate them. Consequently Duplicati ran into a unique-constraint-error.

What happened? Sometime in June
https+++www.nzz.ch.
was added by Firefox to the already existing
https+++www.nzz.ch
I don’t know, whether this was an error by Firefox or by www.nzz.ch. From then on, Duplicati failed with the unique-constraint-error.

For those who didn’t follow the “Exclude invalid Windows paths” link, that plan got superseded by a fix to tolerate such paths. The fix is in 2.0.5.109 Canary. It seems generally fine, but has one issue fixed so far.

Release: 2.0.5.109 (canary) 2020-08-07

  • Improved handling of problematic filenames on Windows and increased test coverage, thanks @dferreyra

and the description of the found-and-fixed-in-next-Canary issue follows that. Eventually this will be in Beta.

I have encountered this error today for the first time. Everything ran fine for months.

I have tried Everything with the query you supplied, but it did not return any results. How could I go about solving this issue/finding the offending folders?

I am running 2.0.5.1_beta_2020-01-18 (which I believe is the latest?)

edit: okay, I found a folder in the backup-scope with a dot at the end (just clicked through it, and saw it).
How can I remove the folder? Bash seems not to be working for me unfortunatly…

edit 2: I got rid of the folder. luckily it was in the cookie-folder of firefox (same as the one Ales-at-gthb mentioned). I was able to get rid of the folder by deleting the cookies for the offending domein from within Firefox itself. Then I ran the backup again, and it went through with a warning ("[Warning-Duplicati.Library.Main.Operation.Backup.UploadSyntheticFilelist-MissingTemporaryFilelist]: Expected there to be a temporary fileset for synthetic filelist (298, duplicati-[random].dindex.zip.aes), but none was found?"). I ran it again immediatly and it ran without error or warning. I think the warning was a remnant of the failed run earlier.

At the start of a backup after an interrupted backup, there’s supposed to be a synthetic filelist based on the backup before the interrupted backup, amended by the interrupted backup progress before its interruption.
There was a bug. It shouldn’t be talking about dindex file at all but dlist. This is fixed in Canary but not Beta.

You write “Bash seems not to be working for me unfortunatly”. Can you be more specific? OK, the error occurs on Windows, where there is no bash pre-installed. You need to installed it. I recommend cygwin.

Then the cookie-folder of firefox seems to be problematic. Only one false setting of a cookie of some website is enough to let your Duplicati backups fail.

Open your bash-prompt and go into this directory:
cd /cygdrive/c/Users/<your username goes here>/AppData/Roaming/Mozilla/Firefox/Profiles/<your firefox profile goes here>.default-release/storage/default

This command lists all directories in the cookie folder:
ls -d *

Look for similar ones like those:
https+++www.nzz.ch
https+++www.nzz.ch.
Windows Explorer cannot differentiate those two. bash can.

Delete one of them using rm:
rm -r https+++www.nzz.ch
…and your problem is gone.

And as soon as a new version of Duplicati is released, your problem is really gone and you can delete cygwin :slightly_smiling_face:

Yeah, sorry about the non-specific indication of the error installing bash. I tried installing it using some instructions I Googled: https://altis.com.au/installing-ubuntu-bash-for-windows-10/

I could enable the developer options, and check the “Windows subsystem for Linux”, but after rebooting the bash-command was not recognized like the instructions said. I did not delve further into this since I resolved my problem by deleting the cookies through Firefox as indicated in my edit of the post.

Although I think letting Firefox do the delete was less risk, you can manipulate these troublesome files in a Command Prompt by putting \\?\ in front of the path you want. This avoids Windows’ path normalization.