Hash Mismatch on dindex files

Windows Server 2012
Duplicati 2.0.4.23_beta_2019-07-14

Backup is to external USB Storage. Ran into I/O issue last week. Ran a full verify Error reports hash mismatch on 2 dindex files which are associated with the First Backup. Backup is still running but reporting this error each time.

As these are index files what is the procedure to regenerate them. Their associated dblock files are still intact and available.

I am also interested in finding out.

I did an extreme test and deleted ALL dindex files. The database recreation afterward succeeded but of course took a while. Unfortunately, no dindex files were recreated. I understood from some past threads that running a repair should fix missing dindex files.

I suspect some dindex files are corrupt on a couple of my production backups and would like to fix them.

I just tried the delete-dindex-and-run-repair on small 2.0.4.5 and 2.0.4.30 backups, and dindex came back. These were new backups, and presumably had good DB data. Without that, making dindex may not work.

You could try increasingly more complicated and long-term backups to look for when dindex recreates fail.

I’m not sure how to fix bad DB data, if that’s the issue. You could see what happens on a test backup after deleting all dindex files and doing a Recreate then doing a Repair. Will you get no dindex or will you get all?

In my case I suspect dindex corruption on one of my backups because during a full database recreate Duplicati decides it needs to download some of the dblock files. From previous discussions this seems to happen if the dindex files are not “complete.”

So it isn’t DB corruption per se, because I’m doing a full recreate.

Interesting thought - I didn’t even try to do a Repair after Recreate. I assumed that Recreate was basically a Delete+Repair. I’ll give it a shot…

I think it is, but Repair puts split modes into one command:

If no local database is found or the database is empty, the database is re-created with data from the storage. If the database is in place but the remote storage is corrupt, the remote storage gets repaired with local data (if available).

MissingVolumes fix is in RunRepairRemote, which looks like it isn’t run when the database is not found:

Just completed a test on a small backup.

  • Deleted all dindex files
  • Ran database Recreate
  • Database recreated successfully, no dindex files present yet
  • Ran database Repair
  • Finished almost instantly, no dindex files present yet
  • Ran a backup job
  • Duplicati then reported the missing dindex files and recreated them

So they aren’t recreated until you actually run a backup job, it seems. Good to know!

I’m going to test this later today on the backup set where I suspect dindex corruption. If all goes well, hopefully the dindexes will be recreated properly so that future database “recreation” will be much faster and not require dblock downloads.

Ran a variation of the above test where I do not recreate database at second step.
Duplicati behaves differently and recreates the dindex files at the Repair process.

  • Deleted all dindex files
  • Attempted to run job, received failure about missing files on back end
  • Ran repair and dindex files were recreated
1 Like

why do the dindex files exist if they’re unnecessary? i was able to delete them from my backup to cut the number of backup files in half and was still able to restore

I’m not an expert on this, but from what I understand the dindex files show which blocks are found in which dblock files.

If you delete the dindex files, you may not notice any issue because your local database still has that information. The dindex files will be regenerated and reuploaded at your next backup from my experience above.

If you delete the dindex files AND your local database, recreating your local database requires rescanning all dblock files, a very time consuming process.

In short, I wouldn’t delete them.

Is file count a worry? The dindex files are far smaller than their dblocks, so space shouldn’t be a worry.

Agree completely, for reasons given. Even without intentional DB delete, what if your drive/system fails?
Direct restore from backup files builds a partial temporary database, which is far faster with dindex files.

Technical background:
How the backup process works
How the restore process works