Failed to read local db, Duplicati blocks itself access to it

[rodrigo_vda]

A few days ago my Backup job stopped working, when going to the live Verbose log all I could find is:

Failed to read local db C:\Windows\system32\config\systemprofile\AppData\Local\Duplicati\ZYEDFWQRGR.sqlite, error: database is locked database is locked

{"errorCode":"Busy","ClassName":"System.Data.SQLite.SQLiteException","Message":"database is locked\r\ndatabase is locked","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   en System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)\r\n   en System.Data.SQLite.SQLiteDataReader.NextResult()\r\n   en System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)\r\n   en System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)\r\n   en Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64(IDbCommand self, Boolean writeLog, String cmd, Int64 defaultvalue, Object[] values)\r\n   en Duplicati.Library.Main.Database.LocalDatabase..ctor(IDbConnection connection, String operation)\r\n   en Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":"8\nStep\nSystem.Data.SQLite, Version=1.0.111.0, Culture=neutral, PublicKeyToken=db937bc2d44ff139\nSystem.Data.SQLite.SQLite3\nBoolean Step(System.Data.SQLite.SQLiteStatement)","HResult":-2147481681,"Source":"System.Data.SQLite","WatsonBuckets":null}

I am running Duplicati as a service:

I did some digging and I saw that it was Duplicati itself the one locking the database. I didn’t know what to do, so I tried this:

1. Forcefully kill all Duplicati processes
2. Renaming the database in C:\Windows\System32\config\systemprofile\AppData\Local\Duplicati to <database name>.sqlite.bak
3. Relaunching the service
4. Going to the web GUI and launch a Repair job

Same error appears.

The next thing I saw was that, for some reason, when launching the service, then there’s two "Duplicati.WindowsService.exe" executables, from:

• "C:\Program Files\Duplicati 2"
• "C:\Program Files\Duplicati 2\updates\2.0.7.1"

meanwhile, "Duplicati.Server.exe" is being launched from:

• "C:\Program Files\Duplicati 2\updates\2.0.7.1"

I suspect one of them is locking the database. I test this by killing the one in "C:\Program Files\Duplicati 2" (the one that is executed as a service) while letting the other one running, and indeed, now Duplicati can rebuild the database without an issue.

This seems weird to me, if the service runs the executable inside the main folder, why does it execute a “mirror” from the update folder? if that’s not wrong behaviour, why does it block access to the database?

Why is this happening?
How do I prevent this from happening again in the future? (for instance, whenever I reboot the computer, it will presumably happen again)
shouldn’t the Duplicati update be applied to the base folder instead of being in a subfolder?

Thank you for your time and assistance

[ts678]

@rodrigo_vda

then there’s two "Duplicati.WindowsService.exe" executables

@ts678

Sounds normal, as many Duplicati processes appear in pairs with the original one (parent) being started from some location (perhaps C:\Program Files\Duplicati 2). That starts the newest version wherever it is.

[SOLVED] Is it ok that I see 5 processes of Duplicati in Windows Task manager? shows a normal full set.

@rodrigo_vda

shouldn’t the Duplicati update be applied to the base folder instead of being in a subfolder?

No. This would prevent unattended updates for less powerful users without right permissions.
Whether you consider that a feature or a bug is up to you, and far-future release may change.
Replace auto updater #4542

[ts678]

@rodrigo_vda

Forcefully kill all Duplicati processes

A cleaner starting point would have been to stop the service, then see if anything was then left over.
I forget whether Duplicati or Windows tends to restart things after manual kills. Basically avoid them.

@rodrigo_vda

Now it has worked for a few days but a new problem has arisen (I think it’s unrelated since it involves access to the database).

I already opened a different thread for that. I’m sorry for causing so much trouble!

Is this a new thread for trying to recover from the very hazardous manual deletion of destination files?
If not, where is this starting from? The first line sounds like backup just broke without previous breaks.

EDIT:

If this turns out to be a continuation topic, and first issue was backup failing from a conflict, did you set
auto-cleanup? Though it’s not visible here in stack trace, I mention it as there’s a known issue, and fix.

[gpatel-fr]

Well, that’s how it is supposed to work. Updates are applied in another directory, and the (original) instance launches the updated version. This is a way to allow self-updating while the application is running. So the original version never opens the database, it’s downgraded to be just a launcher as soon it detects a newer version.

Since you use the Sysinternals utilities, you can use Handle to see which process has opened the file.

[rodrigo_vda]

@ts678

Is this a new thread for trying to recover from the very hazardous manual deletion of destination files?
If not, where is this starting from? The first line sounds like backup just broke without previous breaks.

It’s not. I managed to recover from that, and for a few days it has been doing backups without an issue. Now this has arisen. These two things happened so close in time that they might be related, but I don’t see how, since I have successfully ran backups since the recovery.

@ts678

No. This would prevent unattended updates for less powerful users without right permissions.
Whether you consider that a feature or a bug is up to you

The only trouble I got with this is that I would need to update firewall rules each time Duplicati updates itself, but I’m glad to see this is not an issue.

@gpatel-fr

Since you use the Sysinternals utilities, you can use Handle to see which process has opened the file.

Now I’m rebuilding the database sucessfully, once it finishes I’ll reboot the computer to “force” it a “default” start of the service and see if the issue persists.
I will keep troubleshooting and I’ll come back with the results, either “it fixed itself and I don’t know why” or more details on how to recreate it.

Thank you all for your time and effort.

EDIT:

@ts678

If this turns out to be a continuation topic, and first issue was backup failing from a conflict, did you set
auto-cleanup? Though it’s not visible here in stack trace, I mention it as there’s a known issue, and fix.

It’s not a continuation topic, but just in case it turns out they are related, yeah, It’s set. Just in case too, these are my relevant backup advanced options (send-mail options omitted):

--snapshot-policy=Required
--backup-test-percentage=2
--symlink-policy=Store
--asynchronous-concurrent-upload-limit=1
--auto-cleanup=true
--auto-vacuum=true

[Jojo-1000]

@rodrigo_vda

It’s not a continuation topic, but just in case it turns out they are related, yeah, It’s set

Auto cleanup is broken and will always result in database is locked errors when it tries to repair. You should disable it until the fix is released.

[ts678]

@Jojo-1000

when it tries to repair

Is there a path to this bug here? I know backup can do it, but this looks more like repair-as-recreate. Regardless, the original author felt auto-cleanup enough of a risk that the default was chosen as off.

duplicati/duplicati/blob/44ce6d746f6784bac8a3d4e65b439d8b1ccd2db8/Duplicati/Library/Main/Operation/BackupHandler.cs#L159-L165

For the spot above, one could probably About → Show log → Live → Verbose to see if it was tried, however it sounds like whatever was going wrong went away, so watching it repeat may be difficult.
Original post log findings start in the repair handler, and possibly was just from backup. Don’t know.

@rodrigo_vda

1. Going to the web GUI and launch a Repair job

Same error appears.

got into some confusion on processes, and I’m still not sure whether there was anything else running. Scheduled backup might have started instantly, so then “launch a Repair job” would likely just queue.
Status line would give some incorrect information (it calls everything “backup”), but maybe add clues.

Fix database is locked when performing auto cleanup #4971 (thanks!) helps backup, but is there one?
Thanks too for the PR-in-queue to get more info in the job log. The server log is off the standard path.

@rodrigo_vda

I will keep troubleshooting and I’ll come back with the results, either “it fixed itself and I don’t know why” or more details on how to recreate it.

For further digging (if desired), look at server log About → Show log → Stored as well as the job logs.
If history can’t be reconstructed, I would say turn option off anyway, and see if everything stays OK…