Backups fail, missing files, try to delete but "Database file does not exist"

1

I’ve been struggling with a corrupt backup for 3 months. The backup sets are go back several years, so I would like to keep those if I can instead of deleting and starting over from scratch.

About 3 months ago I got an errors on my backups that several files were missing from an older backup version. I tried repair database, rebuild files, recreate database, etc., but nothing worked. Ultimately, I deleted that backup version. Backups still failed with missing files on a different version. Same thing: repair, rebuild, recreate, ultimately delete.

BTW, the missing file errors are all similar to:

Remote file referenced as duplicati-bbxxxxxxxxxxxxxxxxxxxxxxxxxxxxxb5.dblock.zip.aes by duplicati-ixxxxxxxxxxxxxxxxxxxxxxxxxxxxxxd0.dindex.zip.aes, but not found in list, registering a missing remote file

Again, same thing, missing files on yet a third backup. Repair, rebuild, try to delete. However, when I try to delete this version, I get the error: “Database file does not exist: /config/.config/Duplicati/68##################.sqlite”.

My databases are actually in /config/ (this errant backup is /config/AMxxxxxxxx.sqlite). However, when I run my delete command, it creates /config/.config/Duplicati/dbconfig.json, in which it sets Path to AM… and DatabasePath to 68…

What’s going on? Where did this numbered database originate? My databases are all letters, not all numbers. Most importantly, how do I salvage this backup? I don’t want to loose the previous years worth of backups because of some missing files from a few months ago, but I haven’t been able to backup for months because of the missing file errors. Why can’t Duplicati ignore these errors and continue new backups? If something is missing, fine, backup the existing files, but don’t fail all future backups because of 4 missing files from old backups. I don’t understand. Help!

2

Welcome to the forum @BennTech

What is your current and (if you recall) past Duplicati version, on what OS, and to what storage type?

One helpful dbconfig.json explanation is in Does the AFFECTED command need a local DB? so if it’s causing trouble and you know your database path, then you can just pass it using the –dbpath option.

The file doesn’t have much use in the GUI. AFAIK it’s for command line. Is that how you’re doing this?

The hard question is what began going wrong 3 months ago. Any idea what may have changed then? Have any logs, or other information? Going way back in time to troubleshoot is frequently not possible.

If storage permits this, sometimes people start a new backup just to have a backup running while they see what can be made of the old backup. Using another system is best because attempts take awhile. There are a finite number of tools to use, and you’ve already used some of them but perhaps not all…

A general precaution is to keep backups of the DB (and maybe also backup files) while experimenting.

3

Sorry for the silence I have been working on this and I finally did get it resolved by deleting all the [corrupt/partial] backups from the previous 3 months. For those interested, here’s some additional info.

I’m currently running Duplicati 2.0.4.23_beta_2019-07-14 in a docker on Unraid (Slackware). All my databases are local in /config/. Backups are to a separate NAS but appear as local inside the docker.

3 months ago I added a large chunk of client videos and photos to the backup source (files are mostly static and with only minor changes a few times per year). I don’t know if it was Duplicati’s access to the source or destination, network issues, NAS issues, etc., but Duplicati expected more files in the backup destination. And my troubles began.

Despite the GUI, I normally prefer to do troubleshooting on the command-line. IMHO, it provides better feedback. Unfortunately, I did not keep good records of previous errors, but they were all revolving around the missing file errors mentioned previously.

I tried various commands (repair, recreate, list-broken-files, purge-broken-files, etc.), but could never get the corrupt backup version fixed. Thus, I deleted that backup version. However, then Duplicati complained about the next most recent backup. Troubleshoot…delete…complaints about next backup.

At some point, my delete stopped working. Not sure exactly what changed (I do regularly update my dockers, but Duplicati itself is the same version as when this started), but I started getting the “Database file does not exist: /config/.config/Duplicati/68##################.sqlite” error. I tracked that down to dbconfig.json file. I have no idea why it’s looking in that folder for that database.

Subsequently, I tried modifying the JSON file directly, which got me a little further with hopeful messages about “Listing remote foler …” Unfortunately, these attempts also failed with messages like “ErrorID: FolderMissing The folder /config/AMxxxxxxxx.sqlite does not exist”.

Finally, my solution was to give up with my command line and use the GUI’s Commandline option to delete the corrupt backup versions. For those who are looking to do that:

  1. Select Your Backup > Commandline …
  2. Select Command: delete
  3. Select Add advanced option: version (under Core options)
  4. Enter the version number(s) you want to delete.

In my case, I had to delete all backups after this corruption occurred, which was fine for me since these new files haven’t changed. Hopefully if anyone else has this issue, this is workable solution for you as well.

3 Likes