Recreate constraint violation on DuplicateBlock

[avmaksimov]

@kenkendk

Alternative is to download 2.1.0.106, which has a fix for this situation.

Thank you. I tried this solution but I’ve got the same error:

* 2025-01-14 11:04:15 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b6fb1499584624539b2d92fa6a6301a12.dblock.zip.aes by duplicati-i50c82057dc7d457abf359e1d86249529.dindex.zip.aes, but not found in list, registering a missing remote file
* 2025-01-14 11:04:58 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b77e5568473db4374bc75416f85982a35.dblock.zip.aes by duplicati-i573884406f3b463da9786381ce64d750.dindex.zip.aes, but not found in list, registering a missing remote file
* 2025-01-14 11:07:13 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bc960abec0e514fef8f305d67f3b5c1e7.dblock.zip.aes by duplicati-i63fe0b55e5814313b837b2eec3d66232.dindex.zip.aes, but not found in list, registering a missing remote file
* 2025-01-14 11:09:01 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bb0b40f36949d47238d5fa1aeb2e9388c.dblock.zip.aes by duplicati-icdf43deb183d42f3991f9b2e4feebd13.dindex.zip.aes, but not found in list, registering a missing remote file
* 2025-01-14 11:09:14 +03 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: Сбой при операции Repair с ошибкой: constraint failed UNIQUE constraint failed: DuplicateBlock.BlockID, DuplicateBlock.VolumeID SQLiteException: constraint failed UNIQUE constraint failed: DuplicateBlock.BlockID, DuplicateBlock.VolumeID

If you don’t mind I want to help you in testing a new solution of this problem.

[kenkendk]

@avmaksimov

If you don’t mind I want to help you in testing a new solution of this problem.

I would very much like to recreate this problem. Do you have any idea why it happens?

It looks like you have 4 files that point nowhere, and this seems to trigger it somehow.

[avmaksimov]

May be really problem with local disk for local db but not for backuping data. But at the same time I started to see this problem only after update to the last beta.

May be also some problem with Internet or even problem with electricity when backuping.

I think these 4 reasons could have occurred with equal probability.

[kenkendk]

@avmaksimov

I started to see this problem only after update to the last beta.

That is because I added an index to the database with 2.1.0.2 to fix another issue.
This index prevents the same information from being stored twice, which is what is causing the error message.

I will try again to see if I can recreate the issue with larger sets of duplicated+missing data.

[ts678]

What is Destination Storage type? Does it permit good view, move, and rename of files?

I’m curious about times and sizes of the four bad dindex files, e.g. do they seem similar?

One way to get these files is below:

recreate "Remote file referenced as ..." error after interrupted backup then compact

It might also be good to see how About → Show log → Live → Verbose shows the error.
Although that won’t explain the “why”, it might help to explain then “when” in the process.

@avmaksimov

My error occurred after message about db malfunctions. I recreated DB

What sort of malfunctions? That might be a clue as to what’s going wrong with Recreate.

@kenkendk

@avmaksimov

What should I do? Remove index files?

I guess that could work, but it will be slow.

Why would removing (I prefer a move or change of prefix, in case they’re relevant) slow things? Removing all index files will make Recreate really slow, but are these four files doing any good?

[avmaksimov]

@ts678

What is Destination Storage type? Does it permit good view, move, and rename of files?

Yes, quite good. I connect via webdav. I can do any file operation via web interface. Also approx all backups use this storage. I use it for some years and this is the first time I faced with this problem.

@ts678

I’m curious about times and sizes of the four bad dindex files, e.g. do they seem similar?

Sizes are different for the all files.

@ts678

It might also be good to see how About → Show log → Live → Verbose shows the error.
Although that won’t explain the “why”, it might help to explain then “when” in the process.

• Jan 15, 2025 6:09 PM: Сбой при операции Repair с ошибкой: constraint failed UNIQUE constraint failed: DuplicateBlock.BlockID, DuplicateBlock.VolumeID

• Jan 15, 2025 6:09 PM: Remote file referenced as duplicati-bb0b40f36949d47238d5fa1aeb2e9388c.dblock.zip.aes by duplicati-icdf43deb183d42f3991f9b2e4feebd13.dindex.zip.aes, but not found in list, registering a missing remote file

• Jan 15, 2025 6:07 PM: Remote file referenced as duplicati-bc960abec0e514fef8f305d67f3b5c1e7.dblock.zip.aes by duplicati-i63fe0b55e5814313b837b2eec3d66232.dindex.zip.aes, but not found in list, registering a missing remote file

• Jan 15, 2025 6:05 PM: Remote file referenced as duplicati-b77e5568473db4374bc75416f85982a35.dblock.zip.aes by duplicati-i573884406f3b463da9786381ce64d750.dindex.zip.aes, but not found in list, registering a missing remote file

• Jan 15, 2025 6:04 PM: ExecuteScalarInt64: SELECT “ID” FROM “Remotevolume” WHERE “Name” = “duplicati-ibbb06864ebb545caaa6c28d0054362fa.dindex.zip.aes” took 0:00:00:00.000

• Jan 15, 2025 6:04 PM: Starting - ExecuteScalarInt64: SELECT “ID” FROM “Remotevolume” WHERE “Name” = “duplicati-ibbb06864ebb545caaa6c28d0054362fa.dindex.zip.aes”

• Jan 15, 2025 6:04 PM: ExecuteScalarInt64: SELECT “VolumeID” FROM “Block” WHERE “Hash” = “tt1/TQrRJ/M7xhzv9D9g2fopY6n1vbqBkxvicboV4u8=” AND “Size” = 102400 took 0:00:00:00.000

• Jan 15, 2025 6:04 PM: Starting - ExecuteScalarInt64: SELECT “VolumeID” FROM “Block” WHERE “Hash” = “tt1/TQrRJ/M7xhzv9D9g2fopY6n1vbqBkxvicboV4u8=” AND “Size” = 102400

• Jan 15, 2025 6:04 PM: ExecuteScalarInt64: SELECT “VolumeID” FROM “Block” WHERE “Hash” = “jMGO7wzSjeH7xvUl2rcsLfYS4xIz7VesYvwtf7JzhhM=” AND “Size” = 102400 took 0:00:00:00.000

• Jan 15, 2025 6:04 PM: Starting - ExecuteScalarInt64: SELECT “VolumeID” FROM “Block” WHERE “Hash” = “jMGO7wzSjeH7xvUl2rcsLfYS4xIz7VesYvwtf7JzhhM=” AND “Size” = 102400

• Jan 15, 2025 6:04 PM: ExecuteScalarInt64: SELECT “VolumeID” FROM “Block” WHERE “Hash” = “3U2Ft+aZDhIo0WSQizBhxvtVIBCIWlRj8XSXhptBLxQ=” AND “Size” = 102400 took 0:00:00:00.000

• Jan 15, 2025 6:04 PM: Starting - ExecuteScalarInt64: SELECT “VolumeID” FROM “Block” WHERE “Hash” = “3U2Ft+aZDhIo0WSQizBhxvtVIBCIWlRj8XSXhptBLxQ=” AND “Size” = 102400

• Jan 15, 2025 6:04 PM: ExecuteScalarInt64: SELECT “VolumeID” FROM “Block” WHERE “Hash” = “oVkSqdzuWooJ2GwniX7BQcjoUGOaBgliPLaxjLgOPqk=” AND “Size” = 102400 took 0:00:00:00.000

Sorry. I don’t remember. Something about disk may be…

@ts678

Why would removing (I prefer a move or change of prefix, in case they’re relevant) slow things? Removing all index files will make Recreate really slow, but are these four files doing any good?

I see the question is not for me but I can do it. But may be it’s not a good idea for finding the problem.

[ts678]

@avmaksimov

But may be it’s not a good idea for finding the problem.

Probably best to wait for feedback from developer, but good to know it’s easy to play with.
Possibly this needs the exact right mix of that sort of error, along with some that are good.

I don’t know how long the next try at a fix attempt will take. How hard is downtime on you?

@avmaksimov

Sorry. I don’t remember. Something about disk may be…

That must be the answer to “db malfunctions”. It may or may not be relevant to the issues.

[avmaksimov]

@ts678

How hard is downtime on you?

Only God knows. Now - it’s not a problem. But after a failure with data I will not have any backuped data to restore (((.
You helped me with the idea, what I’ll make copy of backuped folder and backup. So It’s not a big problem for waiting.

[kenkendk]

@avmaksimov

Now - it’s not a problem. But after a failure with data I will not have any backuped data to restore (((.

I am pretty sure it can be restored by removing the .dindex files you see in the logs.

If not, you can restore with the RecoveryTool, which does not use the .dindex files at all.

@avmaksimov

You helped me with the idea, what I’ll make copy of backuped folder and backup. So It’s not a big problem for waiting.

Thanks! I have managed to reproduce the error by running the test I made twice. Not sure yet why it needs to be there twice, but since it is now reproducible, I am confident a fix will be out soon.

[kenkendk]

@avmaksimov

You helped me with the idea, what I’ll make copy of backuped folder and backup. So It’s not a big problem for waiting.

Thanks for waiting, I found the cause for this and made a new version 2.1.0.107 that should fix it.

The details for this was slightly different. You need to have:

i1 -> b1 (missing b1)
i2 -> b2
i3 -> b3

Each of b1, b2, and b3 must reference the same data block.

If you have such a setup, the order of files needs to first process one index file where the block file is missing. This causes the recreate to assign the blocks to the missing file (b1).

Later, when reading i2 & i3, these will be recorded as duplicate blocks. After this, Duplicati will attempt to fix this by using either b2 or b3 instead of the missing b1.

But since there are now two sources to pick from, it would try to use both sources, causing a constraint violation. With the update, it picks just one of them.

[avmaksimov]

Thank you very much for solving the problem and describing the algorithm.

I’ve downloaded and tested the new version. Recreating database finished with success with text:

Warnings 3

* 2025-01-20 16:45:44 +03 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 4 missing volumes; attempting to replace blocks from existing volumes
* 2025-01-20 17:19:08 +03 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 4 missing volumes; attempting to replace blocks from existing volumes

Errors 4

* 2025-01-20 15:48:43 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b6fb1499584624539b2d92fa6a6301a12.dblock.zip.aes by duplicati-i50c82057dc7d457abf359e1d86249529.dindex.zip.aes, but not found in list, registering a missing remote file
* 2025-01-20 15:56:16 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b77e5568473db4374bc75416f85982a35.dblock.zip.aes by duplicati-i573884406f3b463da9786381ce64d750.dindex.zip.aes, but not found in list, registering a missing remote file
* 2025-01-20 16:24:17 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bc960abec0e514fef8f305d67f3b5c1e7.dblock.zip.aes by duplicati-i63fe0b55e5814313b837b2eec3d66232.dindex.zip.aes, but not found in list, registering a missing remote file
* 2025-01-20 16:44:25 +03 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bb0b40f36949d47238d5fa1aeb2e9388c.dblock.zip.aes by duplicati-icdf43deb183d42f3991f9b2e4feebd13.dindex.zip.aes, but not found in list, registering a missing remote file

But the next backup finished with success. Also later I made repair and it finished with success without neither errors nor warnings.

So I think you’re a wizard) And the problem I was solved.

[kenkendk]

@avmaksimov

I’ve downloaded and tested the new version. Recreating database finished with success with text:

Thanks so much for confirming!

It looks exactly as expected, but perhaps we should downgrade the MissingFileDetected messages to warnings.