Missing .dindex.aes files on the remote

Support
1

I have some missing files ‘.dindex.aes’ on the remote. The remote is a SMB volume configured with the ‘local file’ backend, and the ‘\server\share\directory’ target location. The Duplicati Host is Windows 10.

I don’t exactly know how the .dindex files got removed, maybe Duplicati crashed while updating them, but they are actually not on the remote drive anymore. Only four files are missing, none of the .dblock.aes files are reported missing, and I can restore. However, the automatic backup fail and are not committed.

I found a number of similar threads on this forum, I tried some of the proposed solutions (rebuild the DB with delete&repair), I used compact etc, nothing removes the error about these missing .dindex.

My understanding is that since all .dblock files are still present, it should be possible to recreate these .dindex files even though it may be potentially pretty long to do so, but I cannot wrap my head about how to do this. Please advise.

2

Missing dindex files can be rebuilt and reuploaded by Duplicati if you use the Repair option. It uses information in the database to rebuild them.

Unfortunately it sounds like you used the Recreate option (delete & repair) so you can’t do that quick fix. That being said, I think a full database recreate should work, it will just take a lot longer as you noted.

When you did try the full database recreate, what was the end result? Did the recreate process complete without error, but you then got a missing dindex error on next backup? Or did the recreate process end in an error?

Also, disregard compact as that is not used to fix issues like this.

3

I did delete&repair, which removes the local database, not the remote, and that didn’t work (the repair says it’s ‘successful’, but subsequent ‘verify’ or ‘backup’ operation still complain about the missing files).

I did not do a full database recreate, the main reason being that I didn’t find how to do it in the documentation. If you can hold my hand I’m willing to try that. I can use the CLI if needed.

4

“Delete & Repair” is the same thing as “Recreate.” Also note that there is no remote database. Just a local database and remote backup files.

If you suspect the local database and remote backup files are out of sync, it’s better to start with the less drastic “Repair” option first. In the case of missing dindex files it could fix that issue really quickly.

It is puzzling to me that the repair option you did succeeded without error but then you get an error when you run your next backup operation. Hmmm

5

My mistake, there is an error when performing the repair operation, see log below.

The remote does contain about 100 or so .dindex.aes files, it is indeed missing the 4 reported files, even after the repair process.

(just for clarity, I was running the beta release until yesterday, I installed the Canary to see if it would help with new bugfixes or something, but that’s not the reason stuff got out of whack).

{
  "MainOperation": "Repair",
  "RecreateDatabaseResults": {
    "MainOperation": "Repair",
    "ParsedResult": "Success",
    "Version": "2.0.6.100 (2.0.6.100_canary_2021-08-11)",
    "EndTime": "2022-02-25T13:59:16.6541261Z",
    "BeginTime": "2022-02-25T00:53:52.1357181Z",
    "Duration": "13:05:24.5184080",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 4006,
      "BytesUploaded": 0,
      "BytesDownloaded": 33682124429,
      "FilesUploaded": 0,
      "FilesDownloaded": 3985,
      "FilesDeleted": 0,
      "FoldersCreated": 0,
      "RetryAttempts": 16,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 0,
      "KnownFileSize": 0,
      "LastBackupDate": "0001-01-01T00:00:00",
      "BackupListCount": 0,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": 0,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Repair",
      "ParsedResult": "Success",
      "Version": "2.0.6.100 (2.0.6.100_canary_2021-08-11)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2022-02-25T00:53:52.1086937Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Error",
  "Version": "2.0.6.100 (2.0.6.100_canary_2021-08-11)",
  "EndTime": "2022-02-25T13:59:17.1385741Z",
  "BeginTime": "2022-02-25T00:53:52.1086937Z",
  "Duration": "13:05:25.0298804",
  "MessagesActualLength": 8018,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 4,
  "Messages": [
    "2022-02-24 19:53:52 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started",
    "2022-02-24 19:53:52 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2022-02-24 19:54:57 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (6.47 KB)",
    "2022-02-24 19:55:10 -05 - [Information-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-RebuildStarted]: Rebuild database started, downloading 35 filelists",
    "2022-02-24 19:55:10 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200423T012148Z.dlist.zip.aes (762.54 KB)",
    "2022-02-24 19:55:10 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200423T012148Z.dlist.zip.aes (762.54 KB)",
    "2022-02-24 19:55:10 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200530T030000Z.dlist.zip.aes (745.29 KB)",
    "2022-02-24 19:55:10 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200530T030000Z.dlist.zip.aes (745.29 KB)",
    "2022-02-24 19:55:11 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200704T040000Z.dlist.zip.aes (2.33 MB)",
    "2022-02-24 19:55:13 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200704T040000Z.dlist.zip.aes (2.33 MB)",
    "2022-02-24 19:55:13 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200809T180000Z.dlist.zip.aes (5.66 MB)",
    "2022-02-24 19:55:13 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200809T180000Z.dlist.zip.aes (5.66 MB)",
    "2022-02-24 19:55:15 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200920T230000Z.dlist.zip.aes (2.23 MB)",
    "2022-02-24 19:55:16 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200920T230000Z.dlist.zip.aes (2.23 MB)",
    "2022-02-24 19:55:25 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20201028T040000Z.dlist.zip.aes (2.29 MB)",
    "2022-02-24 19:55:25 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20201028T040000Z.dlist.zip.aes (2.29 MB)",
    "2022-02-24 19:55:27 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20201129T050000Z.dlist.zip.aes (3.16 MB)",
    "2022-02-24 19:55:27 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20201129T050000Z.dlist.zip.aes (3.16 MB)",
    "2022-02-24 19:55:29 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20210114T145338Z.dlist.zip.aes (3.29 MB)",
    "2022-02-24 19:55:30 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20210114T145338Z.dlist.zip.aes (3.29 MB)"
  ],
  "Warnings": [],
  "Errors": [
    "2022-02-24 20:05:53 -05 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-IndexFileProcessingFailed]: Failed to process index file: duplicati-idfe60924016843f3a7c2871fdff971f3.dindex.zip.aes",
    "2022-02-24 20:06:35 -05 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-IndexFileProcessingFailed]: Failed to process index file: duplicati-ia618e56be942405ab049eb9f47a59c1f.dindex.zip.aes",
    "2022-02-24 20:07:16 -05 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-IndexFileProcessingFailed]: Failed to process index file: duplicati-i81b20ef108e141ca860e2b8ef58b2135.dindex.zip.aes",
    "2022-02-24 20:07:58 -05 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-IndexFileProcessingFailed]: Failed to process index file: duplicati-i058b0e20d4fd47788eda0703b37f5261.dindex.zip.aes"
  ],
  "BackendStatistics": {
    "RemoteCalls": 4006,
    "BytesUploaded": 0,
    "BytesDownloaded": 33682124429,
    "FilesUploaded": 0,
    "FilesDownloaded": 3985,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 16,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 0,
    "KnownFileSize": 0,
    "LastBackupDate": "0001-01-01T00:00:00",
    "BackupListCount": 0,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": 0,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Repair",
    "ParsedResult": "Success",
    "Version": "2.0.6.100 (2.0.6.100_canary_2021-08-11)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2022-02-25T00:53:52.1086937Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}
6

I tried reproducing your problem.

I deleted a dindex file and then tried to run a backup. As expected, Duplicati complained about a missing file and asked me to run Repair. I ran a regular Repair and the dindex file was regenerated. No errors after that.

For the second test I deleted a dindex file again and this time did the Recreate option (which is what I think you did). Recreate finished but the dindex file was not yet placed on the back end. The file wasn’t placed there until the start of the next backup operation.

So… I would think Repair or Recreate should have fixed this issue for you.