Sqlite orphans and rebuilding database error

Support
1

So, while doing some server clean up, I noticed the .config/Duplicati folder was 20GB, seemed pretty large for some local database files. I figured that the DB needed compacting or something, then I stumbled on this previous post Orphan sqlite files?, sure enough I have files in there going back to when I started using Duplicati. Based on the information, these look like backups from the time the upgrade required a new DB. We can cleanup, right?

Yep, you guessed it, I made a mistake and cleaned up before verifying that I had copies of the files in that folder first. There were several named <backupid>.backup*, several named, backup <backupid> <timestamp>.sqlite, so I figured I’d remove anything that was older than last year. What I didn’t spot was that there was one named <backupid>.sqlite so when I did rm <backupid>.* I deleted the most recent file.

No worries, right? Right?! There is a magical button that should rebuild this file. Under “backup” > Advanced > Database, there are maintenance options that allow you to rebuild the database when the local and remote files are out of sync, this seems like the option we’d want. When I execute, it fails with “The operation Test has failed with error: The requested folder does not exist. FolderMissingException: The requested folder does not exist”, but doesn’t list what folder doesn’t exist, so I’m not sure what it’s looking for.

{
  "MainOperation": "Test",
  "VerificationsActualLength": 0,
  "Verifications": [],
  "ParsedResult": "Fatal",
  "Interrupted": false,
  "Version": "2.0.7.103 (2.0.7.103_canary_2024-04-19)",
  "EndTime": "2024-09-10T15:19:35.213793Z",
  "BeginTime": "2024-09-10T15:18:54.605038Z",
  "Duration": "00:00:40.6087550",
  "MessagesActualLength": 11,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 1,
  "Messages": [
    "2024-09-10 08:18:54 -07 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started",
    "2024-09-10 08:18:54 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-09-10 08:18:54 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Retrying:  ()",
    "2024-09-10 08:19:04 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-09-10 08:19:04 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Retrying:  ()",
    "2024-09-10 08:19:14 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-09-10 08:19:15 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Retrying:  ()",
    "2024-09-10 08:19:25 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-09-10 08:19:25 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Retrying:  ()",
    "2024-09-10 08:19:35 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-09-10 08:19:35 -07 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Failed:  ()"
  ],
  "Warnings": [],
  "Errors": [
    "2024-09-10 08:19:35 -07 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Test has failed with error: The requested folder does not exist\nFolderMissingException: The requested folder does not exist"
  ],
  "BackendStatistics": {
    "RemoteCalls": 5,
    "BytesUploaded": 0,
    "BytesDownloaded": 0,
    "FilesUploaded": 0,
    "FilesDownloaded": 0,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 4,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 0,
    "KnownFileSize": 0,
    "LastBackupDate": "0001-01-01T00:00:00",
    "BackupListCount": 0,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": 0,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Test",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.103 (2.0.7.103_canary_2024-04-19)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2024-09-10T15:18:54.605042Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

Any idea what folder it is looking for? Any idea how to regenerate the sqlite database so it can correctly list the backups?

2

The TEST command describes some of what operation Test does. I’m not sure why it’s running.

The first thing (I think) is a list of the job Destination. The retry says that destination list is failing.

You can also test what’s typically a list by going to edit Destination and using Test connection.

I’m getting a different message with a bad destination, but regardless, please see if yours works.

If destination is gone, there won’t be any files to recreate a database. Do you recognize the job?

3

Well this has lead to an interesting discovery. I check maybe once or twice a month to verify everything is still going smoothly, no errors, etc. Everything looked good. I checked the S3 bucket and the last backup files were written on 05/24/2024, even though I recall the logs showing success.

So I did some poking, and nice hint above, I tested the destination, and it failed with bucket doesn’t exist. Then I noticed somehow the destination bucket had changed from <bucket name> to <aws_id><bucket_name>, which is not what I had configured it as (since 2017 some time). If I remove the ID, and hit test again, I get a pop up saying I should append the ID to the bucket name. If I say yes, it reformats to the same, and the check fails. If I click no, the test succeeds.

Now I’ve changed the bucket name to not have the AWS ID at the front, it’s happily rebuilding the database. Of course, looking in S3 for the dlist files, it looks like my last dlist update was 05/24/2024, so I have to thing I imagined the logs saying things were successful. Or maybe that was the last day I did an upgrade and something changed the bucket, and I missed something somewhere.