HowTo recreate sqlite databases if PC crashed

Reiner2010 · February 4, 2025, 3:27pm

I have created a backup to have the last two versions of one folder to Azure Blob storage.
Also the restore was working fine. But then I thought what will happen if the PC or the harddisk is defect.
So, I removed the SQLite databases and tried first to restore my folder. This was working fine with --no-local-db if both databases are missing or with --rebuild-missing-database if only one database is missing. But restore does not recreate the local databases.
Therefore, I started the backup commandline again and got this error.

Backup started at 04.02.2025 15:53:38
Checking remote backup …
Listing remote folder …
Extra unknown file: duplicati-20250128T104726Z.dlist.zip.aes
Extra unknown file: duplicati-20250128T105226Z.dlist.zip.aes
Extra unknown file: duplicati-20250128T125825Z.dlist.zip.aes
Extra unknown file: duplicati-b0355bb884a8e495c82c7630f7b24b66c.dblock.zip.aes
Extra unknown file: duplicati-b8236a28fad41455790187dae6d80fbb5.dblock.zip.aes
Extra unknown file: duplicati-i4c2ce11e0d5f4aeb8afb7a8b9f78ed45.dindex.zip.aes
Extra unknown file: duplicati-ifeeee60252fd4616a159f8a86f7b7c07.dindex.zip.aes
Die Operation Backup ist mit folgenden Fehler fehlgeschlagen: Found 7 remote files that are not recorded in local storage, please run repair => Found 7 remote files that are not recorded in local storage, please run repair

Error:

ErrorID: ExtraRemoteFiles
Found 7 remote files that are not recorded in local storage, please run repair

Afterwards, I ran the repair command as advised by the backup command.
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” repair “<my Azure URL” --passphrase=“my passphrase”
Listing remote folder …
Downloading file duplicati-20250128T104726Z.dlist.zip.aes (57,23 KB) …
Downloading file duplicati-20250128T105226Z.dlist.zip.aes (57,15 KB) …
Downloading file duplicati-20250128T125825Z.dlist.zip.aes (57,23 KB) …
Downloading file duplicati-i4c2ce11e0d5f4aeb8afb7a8b9f78ed45.dindex.zip.aes (941 Bytes) …
Downloading file duplicati-ifeeee60252fd4616a159f8a86f7b7c07.dindex.zip.aes (53,34 KB) …

But I didn’t find the server sqlite file and the client sqlite file wasn’t complete.

Then I tried the command list-broken-files, but nothing was listed. Same for the command purge-broken-files

The question is now, how can I recreate the needed sqlite files to get the backup process running again?

ts678 · February 4, 2025, 4:51pm

I’m confused by what’s GUI and what’s CLI here. The DB names aren’t given, adding to that.

If GUI, you have a Duplicati-server.sqlite DB with the config, and random-letters.sqlite for jobs.

The use of “both” and “one” isn’t too clear, and I can’t find any --rebuild-missing-database.
I’m not on CLI much, but are you sure you gave that name right? Can you find it in help text?
FWIW Google search can’t find it either, in Duplicati context.

Maybe you mean Duplicati-server.sqlite for GUI’s DB and random-letters.sqlite for the job DB?

If the job DB isn’t complete, repair run should have giving you some warning or error. Did it?

These need a job database, and maybe you don’t have one, however I find this difficult to follow.

Are you running GUI job at all? That’s what Duplicati-server.sqlite is for. To recreate that DB, the Import and export backup configurations could be used, or ServerUtil can export all jobs at once.

Regardless, put exported jobs somewhere safe, otherwise you’ll have to guess at old GUI config.

If you’re stricly CLI, this doesn’t apply, as there’s no Duplicati-server.sqlite, likely just CLI options.

If you’re GUI, after import of saved configs, you can go to Database screen and click on Repair.
Something might go wrong, as it might have here, but manual error is less likely. Need messages either way, so maybe you can clarify some of the things asked about here, and procedures used.

Reiner2010 · February 5, 2025, 7:42am

OK, I will be more precise to explain the scenario. I use Duplicati on Windows and use the commandline for this purpose.

In %LOCALAPPDATA%\Duplicati
there is a subfolder called control_dir_v2
and serveral files:

control_dir_v2
dbconfig.json
Duplicati-server.sqlite
installation.txt
IWCSSHHYTM.sqlite
machineid.txt
OEYBZJEXRJ.sqlite
RSIQYOIPTX.sqlite

In %ALLUSERSPROFILE%\Duplicati
is the sqlite DB stored defined by my backup commandline: MyPCName.sqlite.

CommandLine Backup:
duplicati.commandline backup “azure://azureContainer?azure-account-name=azureAccount&azure-access-key=myAccessKey” C:\MyFolder\ –passphrase=“MyPassPhrase” --dbpath=“C:\ProgramData\Duplicati\MyPCName.sqlite” --backup-name=“BackupMyFolder” --backup-id=“MyPCName” --exclude-files-attributes=temporary --disable-module=console-password-input

CommandLine Restore:
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” restore “azure://azureContainer?azure-account-name=azureAccount&azure-access-key=myAccessKey” --passphrase=“MyPassPhrase” --dbpath=“C:\ProgramData\Duplicati\MyPCName.sqlite” --restore-path=C:\MyFolder\ –backup-name=“BackupMyFolder” --backup-id=“MyPCName” --disable-module=console-password-input --version=0

Both commandlines work well, but I thought what will happen if a sqlite DB got corrupt or if my harddisk is defect.

For that reason I moved the content of %LOCALAPPDATA%\Duplicati and %ALLUSERSPROFILE%\Duplicati to a save place and started the restore, with the command.

CommandLine Restore --no-local-dbconfig:
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” restore “azure://azureContainer?azure-account-name=azureAccount&azure-access-key=myAccessKey” --passphrase=“MyPassPhrase” --dbpath=“C:\ProgramData\Duplicati\MyPCName.sqlite” --restore-path=C:\MyFolder\ –backup-name=“BackupMyFolder” --backup-id=“MyPCName” --disable-module=console-password-input --version=0 --no-local-dbconfig

The restore worked well, but if I then try to start my backup commandline again, I got the error as reported above.

I did also tests, if only the content of one folder is missing. Either %LOCALAPPDATA%\Duplicati OR %ALLUSERSPROFILE%\Duplicati and used the following command for it.

CommandLine Restore --rebuild-missing-database:
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” restore “azure://azureContainer?azure-account-name=azureAccount&azure-access-key=myAccessKey” --passphrase=“MyPassPhrase” --dbpath=“C:\ProgramData\Duplicati\MyPCName.sqlite” --restore-path=C:\MyFolder\ –backup-name=“BackupMyFolder” --backup-id=“MyPCName” --disable-module=console-password-input --version=0 --rebuild-missing-database

Also in this case the restore worked and C:\MyFolder\ was restored, but the sqlite DB(s) not.

The question was then how to recover the sqlite databases, if I would have this problem.
If you then execute the backup commandline it is telling you, you have to run the repair.

CommandLine Repair:
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” repair “azure://azureContainer?azure-account-name=azureAccount&azure-access-key=myAccessKey” --passphrase=“MyPassPhrase”

That is the current situation, that I enforced this issue to see how to deal with it, but right now, with no success.

This is only a simulation of an issue, but in case this would happen I would like to know how to recover my databases, so that the backup commandline will work again.

My question is what is the recommended approach here to get the sqlite databases back?

ts678 · February 5, 2025, 1:10pm

I don’t think that option exists. Maybe this was --no-local-db, with help:

–no-local-db (Boolean): Disable the local database
When listing contents or when restoring files, the local database can be skipped. This is usually slower, but can be used to verify the actual contents of the remote store.
* default value: false

which may explain why you didn’t get a local database. Code actually says:

github.com

duplicati/duplicati/blob/c6e3c20391791024d9117525df8bb33e2e702abb/Duplicati/Library/Main/Operation/RestoreHandler.cs#L116-L123


      
          if (!m_options.NoLocalDb && SystemIO.IO_OS.FileExists(m_options.Dbpath))

          {

              db = new LocalRestoreDatabase(m_options.Dbpath);

              db.SetResult(m_result);

          }

          else

          {

              Logging.Log.WriteInformationMessage(LOGTAG, "NoLocalDatabase", "No local database, building a temporary database");

To test, you could add --console-log-level=information to see if says “temporary database”.

Probably meaning “Extra unknown file” because saying --no-local-db built only temporary DB.
Empty DB at your --dbpath doesn’t know about any remote files, so gives the “Extra unknown”.

Giving both --dbpath (saying use the local db) and also --no-local-db seems a contradiction.

I’m not on CLI much, but your disaster recovery restore sounds a lot like what GUI describes as

Restoring files if your Duplicati installation is lost

If you don’t have a configuration file, you have to supply all needed information yourself. Select Direct restore from backup files … and click Next.

CLI never has a GUI-style configuration file, but maybe you ran CLI version of a “Direct restore”.

Support temporary backup configurations for restore purposes #5933 request info explains more.

Direct restore from backup files has the same problem you ran into, which is it’s a temporary DB.
To make a permanent DB for a job (this may change if above enhancement gets done someday):

Repair

If the local database is missing, it is recreated from the remote storage.

Since you use a custom --dbpath, be sure to provide that so that database is in the usual place.

ts678 · February 5, 2025, 1:37pm

Add option to automatically create database if none exists #5932

In the current version, running a backup when no local database exists causes an error.

enhancement idea is also relevant, except I’m not sure how one would know to use option.

kenkendk · February 5, 2025, 7:55pm

If you want to recreate the database (for testing in this case), you can use repair instead:

“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” repair 
“azure://azureContainer?azure-account-name=azureAccount&azure-access-key=myAccessKey” 
--passphrase=“MyPassPhrase” 
--dbpath=“C:\ProgramData\Duplicati\MyPCName-rebuilt.sqlite”

Since MyPCName-rebuilt.sqlite does not exist, it will create it.
If the database exists, it will instead try to repair the remote storage.

Not sure where you got --rebuild-missing-database from, it is not listed as a supported option?

Reiner2010 · February 6, 2025, 5:27am

Thank you, this worked for me.