How to fix missing volumes?

[babelfish]

I wanted to try out/validate my remote backup. So I deleted (moved to different folder) my local db and tried to recreate the database. Unfortunately I got the following errors (full log at the end):

"Warnings": [
    "2024-01-27 16:08:01 +01 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 3 missing volumes; attempting to replace blocks from existing volumes",
    "2024-01-27 16:13:29 +01 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 3 missing volumes; attempting to replace blocks from existing volumes"
  ],
  "Errors": [
    "2024-01-27 16:03:44 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b1eb5e3061c1a45a3839a4bc47d5a144b.dblock.zip.aes by duplicati-i35985736e69e4faea5a5bc8ce7b6ea6f.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2024-01-27 16:07:07 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b1512b5fb010d484d864010b5ac8556f9.dblock.zip.aes by duplicati-id48eceb4bdd44bf9a28901bdd165ed95.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2024-01-27 16:07:36 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b432335a1483543889dd9ad45cb187060.dblock.zip.aes by duplicati-ie9ca759663894befa58f2246dce72053.dindex.zip.aes, but not found in list, registering a missing remote file"
  ]

Running the command line with affected didn’t give me any results:

Duplicati.CommandLine.exe affected  "file://D:\\duplicati backup test\\" duplicati-b1eb5e3061c1a45a3839a4bc47d5a144b.dblock.zip.aes duplicati-b1512b5fb010d484d864010b5ac8556f9.dblock.zip.aes duplicati-b432335a1483543889dd9ad45cb187060.dblock.zip.aes --dbpath="C:\WINDOWS\system32\config\systemprofile\AppData\Local\Duplicati\XLAZNTOGUL.sqlite" --passphrase="MY_PASSPHRASE"
No files are affected

The following related log messages were found:
27.01.2024 15:14:34: {"MainOperation":"Repair","RecreateDatabaseResults":{"MainOperation":"Repair","ParsedResult":"Su ...

running list-broken-files and purge-broken-files also didn’t do anything:

"C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe" list-broken-files "file://D:\\duplicati backup test\\" --dbpath="C:\WINDOWS\system32\config\systemprofile\AppData\Local\Duplicati\XLAZNTOGUL.sqlite" --passphrase="MY_PASSPHRASE"
  Listing remote folder ...

"C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe" purge-broken-files "file://D:\\duplicati backup test\\" --dbpath="C:\WINDOWS\system32\config\systemprofile\AppData\Local\Duplicati\XLAZNTOGUL.sqlite" --passphrase="MY_PASSPHRASE"
  Listing remote folder ...

Do you have any idea how this could have happened and how I could fix it?

Here is the full log of the repair/recreate job in case that helps:

            {
  "MainOperation": "Repair",
  "RecreateDatabaseResults": {
    "MainOperation": "Repair",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "2024-01-27T15:14:34.033561Z",
    "BeginTime": "2024-01-27T14:56:08.8883545Z",
    "Duration": "00:18:25.1452065",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 5211,
      "BytesUploaded": 0,
      "BytesDownloaded": 510043858,
      "FilesUploaded": 0,
      "FilesDownloaded": 5210,
      "FilesDeleted": 0,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 0,
      "KnownFileSize": 0,
      "LastBackupDate": "0001-01-01T00:00:00",
      "BackupListCount": 0,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": 0,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Repair",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-01-27T14:56:08.8582191Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Error",
  "Interrupted": false,
  "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
  "EndTime": "2024-01-27T15:14:34.6332821Z",
  "BeginTime": "2024-01-27T14:56:08.8582191Z",
  "Duration": "00:18:25.7750630",
  "MessagesActualLength": 10428,
  "WarningsActualLength": 2,
  "ErrorsActualLength": 3,
  "Messages": [
    "2024-01-27 15:56:08 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started",
    "2024-01-27 15:56:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-01-27 15:56:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (10,13 KB)",
    "2024-01-27 15:57:41 +01 - [Information-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-RebuildStarted]: Rebuild database started, downloading 29 filelists",
    "2024-01-27 15:57:41 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230203T221239Z.dlist.zip.aes (7,08 MB)",
    "2024-01-27 15:57:41 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230203T221239Z.dlist.zip.aes (7,08 MB)",
    "2024-01-27 15:57:41 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230214T154235Z.dlist.zip.aes (7,09 MB)",
    "2024-01-27 15:57:41 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230214T154235Z.dlist.zip.aes (7,09 MB)",
    "2024-01-27 15:57:58 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230228T161739Z.dlist.zip.aes (7,10 MB)",
    "2024-01-27 15:57:58 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230228T161739Z.dlist.zip.aes (7,10 MB)",
    "2024-01-27 15:58:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230313T164542Z.dlist.zip.aes (7,29 MB)",
    "2024-01-27 15:58:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230313T164542Z.dlist.zip.aes (7,29 MB)",
    "2024-01-27 15:58:13 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230413T172901Z.dlist.zip.aes (7,33 MB)",
    "2024-01-27 15:58:13 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230413T172901Z.dlist.zip.aes (7,33 MB)",
    "2024-01-27 15:58:21 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230515T155651Z.dlist.zip.aes (7,38 MB)",
    "2024-01-27 15:58:21 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230515T155651Z.dlist.zip.aes (7,38 MB)",
    "2024-01-27 15:58:29 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230601T161142Z.dlist.zip.aes (7,40 MB)",
    "2024-01-27 15:58:29 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230601T161142Z.dlist.zip.aes (7,40 MB)",
    "2024-01-27 15:58:37 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230625T142750Z.dlist.zip.aes (7,41 MB)",
    "2024-01-27 15:58:37 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230625T142750Z.dlist.zip.aes (7,41 MB)"
  ],
  "Warnings": [
    "2024-01-27 16:08:01 +01 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 3 missing volumes; attempting to replace blocks from existing volumes",
    "2024-01-27 16:13:29 +01 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 3 missing volumes; attempting to replace blocks from existing volumes"
  ],
  "Errors": [
    "2024-01-27 16:03:44 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b1eb5e3061c1a45a3839a4bc47d5a144b.dblock.zip.aes by duplicati-i35985736e69e4faea5a5bc8ce7b6ea6f.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2024-01-27 16:07:07 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b1512b5fb010d484d864010b5ac8556f9.dblock.zip.aes by duplicati-id48eceb4bdd44bf9a28901bdd165ed95.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2024-01-27 16:07:36 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b432335a1483543889dd9ad45cb187060.dblock.zip.aes by duplicati-ie9ca759663894befa58f2246dce72053.dindex.zip.aes, but not found in list, registering a missing remote file"
  ],
  "BackendStatistics": {
    "RemoteCalls": 5211,
    "BytesUploaded": 0,
    "BytesDownloaded": 510043858,
    "FilesUploaded": 0,
    "FilesDownloaded": 5210,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 0,
    "KnownFileSize": 0,
    "LastBackupDate": "0001-01-01T00:00:00",
    "BackupListCount": 0,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": 0,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Repair",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2024-01-27T14:56:08.8582191Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

[gpatel-fr]

Hello

when you have this result, can the database be used, that is, if you go to restore, do you have files available ? And if you do a backup, are the extra remote files marked as deleted in the log ? - that should be the expected result.

[babelfish]

yes, I can restore files. But it still feels weird to have missing dblock files…

I just did another backup job, but there is nothing written in the log about these files:

            {
  "DeletedFiles": 71974,
  "DeletedFolders": 3530,
  "ModifiedFiles": 0,
  "ExaminedFiles": 1,
  "OpenedFiles": 1,
  "AddedFiles": 1,
  "SizeOfModifiedFiles": 0,
  "SizeOfAddedFiles": 9906984,
  "SizeOfExaminedFiles": 9906984,
  "SizeOfOpenedFiles": 9906984,
  "NotProcessedFiles": 0,
  "AddedFolders": 1,
  "TooLargeFiles": 0,
  "FilesWithError": 0,
  "ModifiedFolders": 0,
  "ModifiedSymlinks": 0,
  "AddedSymlinks": 0,
  "DeletedSymlinks": 3,
  "PartialBackup": false,
  "Dryrun": false,
  "MainOperation": "Backup",
  "CompactResults": {
    "DeletedFileCount": 4,
    "DownloadedFileCount": 0,
    "UploadedFileCount": 0,
    "DeletedFileSize": 706724,
    "DownloadedFileSize": 0,
    "UploadedFileSize": 0,
    "Dryrun": false,
    "VacuumResults": null,
    "MainOperation": "Compact",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "2024-02-01T23:11:40.2746239Z",
    "BeginTime": "2024-02-01T23:11:19.1287431Z",
    "Duration": "00:00:21.1458808",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 13,
      "BytesUploaded": 9049191,
      "BytesDownloaded": 61718375,
      "FilesUploaded": 3,
      "FilesDownloaded": 3,
      "FilesDeleted": 5,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 10368,
      "KnownFileSize": 268293795536,
      "LastBackupDate": "2024-02-02T00:10:23+01:00",
      "BackupListCount": 29,
      "TotalQuotaSpace": 4000785100800,
      "FreeQuotaSpace": 114150756352,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-02-01T23:10:23.6861586Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "VacuumResults": null,
  "DeleteResults": {
    "DeletedSetsActualLength": 1,
    "DeletedSets": [
      {
        "Item1": 5,
        "Item2": "2024-01-01T14:23:38+01:00"
      }
    ],
    "Dryrun": false,
    "MainOperation": "Delete",
    "CompactResults": {
      "DeletedFileCount": 4,
      "DownloadedFileCount": 0,
      "UploadedFileCount": 0,
      "DeletedFileSize": 706724,
      "DownloadedFileSize": 0,
      "UploadedFileSize": 0,
      "Dryrun": false,
      "VacuumResults": null,
      "MainOperation": "Compact",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
      "EndTime": "2024-02-01T23:11:40.2746239Z",
      "BeginTime": "2024-02-01T23:11:19.1287431Z",
      "Duration": "00:00:21.1458808",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null,
      "BackendStatistics": {
        "RemoteCalls": 13,
        "BytesUploaded": 9049191,
        "BytesDownloaded": 61718375,
        "FilesUploaded": 3,
        "FilesDownloaded": 3,
        "FilesDeleted": 5,
        "FoldersCreated": 0,
        "RetryAttempts": 0,
        "UnknownFileSize": 0,
        "UnknownFileCount": 0,
        "KnownFileCount": 10368,
        "KnownFileSize": 268293795536,
        "LastBackupDate": "2024-02-02T00:10:23+01:00",
        "BackupListCount": 29,
        "TotalQuotaSpace": 4000785100800,
        "FreeQuotaSpace": 114150756352,
        "AssignedQuotaSpace": -1,
        "ReportedQuotaError": false,
        "ReportedQuotaWarning": false,
        "MainOperation": "Backup",
        "ParsedResult": "Success",
        "Interrupted": false,
        "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
        "EndTime": "0001-01-01T00:00:00",
        "BeginTime": "2024-02-01T23:10:23.6861586Z",
        "Duration": "00:00:00",
        "MessagesActualLength": 0,
        "WarningsActualLength": 0,
        "ErrorsActualLength": 0,
        "Messages": null,
        "Warnings": null,
        "Errors": null
      }
    },
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "2024-02-01T23:11:40.2746239Z",
    "BeginTime": "2024-02-01T23:10:50.9705079Z",
    "Duration": "00:00:49.3041160",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 13,
      "BytesUploaded": 9049191,
      "BytesDownloaded": 61718375,
      "FilesUploaded": 3,
      "FilesDownloaded": 3,
      "FilesDeleted": 5,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 10368,
      "KnownFileSize": 268293795536,
      "LastBackupDate": "2024-02-02T00:10:23+01:00",
      "BackupListCount": 29,
      "TotalQuotaSpace": 4000785100800,
      "FreeQuotaSpace": 114150756352,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-02-01T23:10:23.6861586Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "RepairResults": null,
  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20231222T123218Z.dlist.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-i5d79e255d8c84b59b882b8de7dcfd9e0.dindex.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-b4e4e672435b24173a7841464aa5697b9.dblock.zip.aes",
        "Value": []
      }
    ],
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "2024-02-01T23:11:41.6506658Z",
    "BeginTime": "2024-02-01T23:11:40.5806926Z",
    "Duration": "00:00:01.0699732",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 13,
      "BytesUploaded": 9049191,
      "BytesDownloaded": 61718375,
      "FilesUploaded": 3,
      "FilesDownloaded": 3,
      "FilesDeleted": 5,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 10368,
      "KnownFileSize": 268293795536,
      "LastBackupDate": "2024-02-02T00:10:23+01:00",
      "BackupListCount": 29,
      "TotalQuotaSpace": 4000785100800,
      "FreeQuotaSpace": 114150756352,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-02-01T23:10:23.6861586Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Success",
  "Interrupted": false,
  "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
  "EndTime": "2024-02-01T23:11:41.6749188Z",
  "BeginTime": "2024-02-01T23:10:23.6851604Z",
  "Duration": "00:01:17.9897584",
  "MessagesActualLength": 39,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": [
    "2024-02-02 00:10:23 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
    "2024-02-02 00:10:41 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-02-02 00:10:46 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (10,13 KB)",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bfe40fee3c00e49f48d73d6e011aab10f.dblock.zip.aes (8,62 MB)",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bfe40fee3c00e49f48d73d6e011aab10f.dblock.zip.aes (8,62 MB)",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i4070e4317efc40a39f30e681e879d8f3.dindex.zip.aes (7,42 KB)",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i4070e4317efc40a39f30e681e879d8f3.dindex.zip.aes (7,42 KB)",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20240201T231023Z.dlist.zip.aes (1,04 KB)",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20240201T231023Z.dlist.zip.aes (1,04 KB)",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 26.01.2024 19:22:14, 26.01.2024 16:39:45, 09.01.2024 14:18:48, 08.01.2024 17:31:06, 01.01.2024 14:23:38, 22.12.2023 13:32:18, 16.12.2023 13:25:53, 09.12.2023 13:00:05, 08.12.2023 18:39:49, 24.11.2023 00:30:10, 18.11.2023 15:11:29, 07.11.2023 20:41:41, 06.11.2023 14:22:59, 07.10.2023 16:47:31, 22.09.2023 23:20:15, 04.09.2023 21:21:11, 17.08.2023 17:30:52, 15.08.2023 01:11:38, 02.08.2023 19:26:40, 08.07.2023 00:04:42, 27.06.2023 16:58:31, 25.06.2023 16:27:50, 01.06.2023 18:11:42, 15.05.2023 17:56:51, 13.04.2023 19:29:01, 13.03.2023 17:45:42, 28.02.2023 17:17:39, 14.02.2023 16:42:35, 03.02.2023 23:12:39",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted: ",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 01.01.2024 14:23:38",
    "2024-02-02 00:10:50 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteRemoteFileset]: Deleting 1 remote fileset(s) ...",
    "2024-02-02 00:11:19 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20240101T132338Z.dlist.zip.aes (8,90 MB)",
    "2024-02-02 00:11:19 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20240101T132338Z.dlist.zip.aes (8,90 MB)",
    "2024-02-02 00:11:19 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: Deleted 1 remote fileset(s)",
    "2024-02-02 00:11:40 +01 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 2 fully deletable volume(s)",
    "2024-02-02 00:11:40 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b5e6ba9577c524103a120d98931b8bcdc.dblock.zip.aes (686,86 KB)"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 13,
    "BytesUploaded": 9049191,
    "BytesDownloaded": 61718375,
    "FilesUploaded": 3,
    "FilesDownloaded": 3,
    "FilesDeleted": 5,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 10368,
    "KnownFileSize": 268293795536,
    "LastBackupDate": "2024-02-02T00:10:23+01:00",
    "BackupListCount": 29,
    "TotalQuotaSpace": 4000785100800,
    "FreeQuotaSpace": 114150756352,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2024-02-01T23:10:23.6861586Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

Do I need to increase the log level?

[gpatel-fr]

@babelfish

"CompactResults": {
    "DeletedFileCount": 4,

It seems that it deleted something still, can you still see the offending files (duplicati-i35985736e69e4faea5a5bc8ce7b6ea6f.dindex.zip.aes for example) on your backend ?

[babelfish]

yes, i35985736e69e4faea5a5bc8ce7b6ea6f.dindex.zip.aes is still available on the backend.

I did the whole procedure again after restoring to the original state (recreate DB, then run a backup), but this time I set backup retention to “keep all backups”, now the backup log is showing

"CompactResults": {
    "DeletedFileCount": 0,

but also

    "2024-02-02 10:33:01 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-b1eb5e3061c1a45a3839a4bc47d5a144b.dblock.zip.aes",
    "2024-02-02 10:33:01 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-b1512b5fb010d484d864010b5ac8556f9.dblock.zip.aes",
    "2024-02-02 10:33:01 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-b432335a1483543889dd9ad45cb187060.dblock.zip.aes",

not sure why this hasn’t happened before… Here the full log:

            {
  "DeletedFiles": 71974,
  "DeletedFolders": 3530,
  "ModifiedFiles": 0,
  "ExaminedFiles": 1,
  "OpenedFiles": 1,
  "AddedFiles": 1,
  "SizeOfModifiedFiles": 0,
  "SizeOfAddedFiles": 9906984,
  "SizeOfExaminedFiles": 9906984,
  "SizeOfOpenedFiles": 9906984,
  "NotProcessedFiles": 0,
  "AddedFolders": 1,
  "TooLargeFiles": 0,
  "FilesWithError": 0,
  "ModifiedFolders": 0,
  "ModifiedSymlinks": 0,
  "AddedSymlinks": 0,
  "DeletedSymlinks": 3,
  "PartialBackup": false,
  "Dryrun": false,
  "MainOperation": "Backup",
  "CompactResults": {
    "DeletedFileCount": 0,
    "DownloadedFileCount": 0,
    "UploadedFileCount": 0,
    "DeletedFileSize": 0,
    "DownloadedFileSize": 0,
    "UploadedFileSize": 0,
    "Dryrun": false,
    "VacuumResults": null,
    "MainOperation": "Compact",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "2024-02-02T09:33:26.1593275Z",
    "BeginTime": "2024-02-02T09:33:05.9801978Z",
    "Duration": "00:00:20.1791297",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 8,
      "BytesUploaded": 9047319,
      "BytesDownloaded": 61595127,
      "FilesUploaded": 3,
      "FilesDownloaded": 3,
      "FilesDeleted": 0,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 10373,
      "KnownFileSize": 268303837873,
      "LastBackupDate": "2024-02-02T10:32:44+01:00",
      "BackupListCount": 30,
      "TotalQuotaSpace": 4000785100800,
      "FreeQuotaSpace": 338139254784,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-02-02T09:32:44.4429794Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "VacuumResults": null,
  "DeleteResults": null,
  "RepairResults": null,
  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20231118T141129Z.dlist.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-ic09b6d879a5042aca9e5d1ff92033a17.dindex.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-b078786a4ce6544eabffeec99c76aed6e.dblock.zip.aes",
        "Value": []
      }
    ],
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "2024-02-02T09:33:27.2790651Z",
    "BeginTime": "2024-02-02T09:33:26.5277599Z",
    "Duration": "00:00:00.7513052",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 8,
      "BytesUploaded": 9047319,
      "BytesDownloaded": 61595127,
      "FilesUploaded": 3,
      "FilesDownloaded": 3,
      "FilesDeleted": 0,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 10373,
      "KnownFileSize": 268303837873,
      "LastBackupDate": "2024-02-02T10:32:44+01:00",
      "BackupListCount": 30,
      "TotalQuotaSpace": 4000785100800,
      "FreeQuotaSpace": 338139254784,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-02-02T09:32:44.4429794Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Success",
  "Interrupted": false,
  "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
  "EndTime": "2024-02-02T09:33:27.2940882Z",
  "BeginTime": "2024-02-02T09:32:44.4429794Z",
  "Duration": "00:00:42.8511088",
  "MessagesActualLength": 22,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": [
    "2024-02-02 10:32:44 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
    "2024-02-02 10:33:00 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-02-02 10:33:01 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (10,13 KB)",
    "2024-02-02 10:33:01 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-b1eb5e3061c1a45a3839a4bc47d5a144b.dblock.zip.aes",
    "2024-02-02 10:33:01 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-b1512b5fb010d484d864010b5ac8556f9.dblock.zip.aes",
    "2024-02-02 10:33:01 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-b432335a1483543889dd9ad45cb187060.dblock.zip.aes",
    "2024-02-02 10:33:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-baa51cbf1fc75432c91713ad77aef927a.dblock.zip.aes (8,62 MB)",
    "2024-02-02 10:33:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-baa51cbf1fc75432c91713ad77aef927a.dblock.zip.aes (8,62 MB)",
    "2024-02-02 10:33:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ie4ba24b7e7ef4909b353361b791ec7b4.dindex.zip.aes (7,42 KB)",
    "2024-02-02 10:33:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ie4ba24b7e7ef4909b353361b791ec7b4.dindex.zip.aes (7,42 KB)",
    "2024-02-02 10:33:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20240202T093244Z.dlist.zip.aes (1,04 KB)",
    "2024-02-02 10:33:05 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20240202T093244Z.dlist.zip.aes (1,04 KB)",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting not required",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (10,13 KB)",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20231118T141129Z.dlist.zip.aes (8,72 MB)",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20231118T141129Z.dlist.zip.aes (8,72 MB)",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-ic09b6d879a5042aca9e5d1ff92033a17.dindex.zip.aes (37,47 KB)",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-ic09b6d879a5042aca9e5d1ff92033a17.dindex.zip.aes (37,47 KB)",
    "2024-02-02 10:33:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b078786a4ce6544eabffeec99c76aed6e.dblock.zip.aes (49,99 MB)"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 8,
    "BytesUploaded": 9047319,
    "BytesDownloaded": 61595127,
    "FilesUploaded": 3,
    "FilesDownloaded": 3,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 10373,
    "KnownFileSize": 268303837873,
    "LastBackupDate": "2024-02-02T10:32:44+01:00",
    "BackupListCount": 30,
    "TotalQuotaSpace": 4000785100800,
    "FreeQuotaSpace": 338139254784,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.7.100 (2.0.7.100_canary_2023-12-27)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2024-02-02T09:32:44.4429794Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

Afterwards I deleted the local DB again and run recreate. But unfortunately I’m still getting the same message about the missing dblocks…

    "2024-02-02 11:04:38 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b1eb5e3061c1a45a3839a4bc47d5a144b.dblock.zip.aes by duplicati-i35985736e69e4faea5a5bc8ce7b6ea6f.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2024-02-02 11:07:02 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b1512b5fb010d484d864010b5ac8556f9.dblock.zip.aes by duplicati-id48eceb4bdd44bf9a28901bdd165ed95.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2024-02-02 11:07:24 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b432335a1483543889dd9ad45cb187060.dblock.zip.aes by duplicati-ie9ca759663894befa58f2246dce72053.dindex.zip.aes, but not found in list, registering a missing remote file"

what does removing file listed as Temporary mean exactly?

[gpatel-fr]

@babelfish

Afterwards I deleted the local DB again and run recreate

This is not supposed to help at this point.

In your place I’ll do a full test - set to ‘All’ the parameter, set the advanced parameters full-remote-verification and full-block-verification to true. A block may be invalid because of a crash during an upload.

[ts678]

@babelfish

what does removing file listed as Temporary mean exactly?

A GitHub Issues post from the original author in 2015 explained it like:

There are the following states:

    Temporary: File has not been attempted uploaded, and can be deleted with no issues
    Uploading: File has been queued for upload, and may or may not exist on server
    Uploaded: File has been uploaded correctly, and is expected to exist on server
    Verified: File has been uploaded and seen in a file listing
    Deleting: File has been queued for deletion, but may exist on server.
    Deleted: File has been deleted and verified missing from a file listing

@babelfish

I wanted to try out/validate my remote backup.

This is a good thing to do. It’s better to spot issues earlier than, say, during disaster recovery.
Is this first such test? If so, please look at the dates of offending files to see how old they are.
Older releases were more likely to be a little off, although I think this look can still be induced.

Sometimes the leftover dindex file is needless, but showing that is difficult without some work.

[babelfish]

@gpatel-fr

In your place I’ll do a full test - set to ‘All’ the parameter, set the advanced parameters full-remote-verification and full-block-verification to true. A block may be invalid because of a crash during an upload.

I did a full test with the following command:
"C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe" test "file://D:\\duplicati backup test\\" all --dbpath="C:\WINDOWS\system32\config\systemprofile\AppData\Local\Duplicati\REKMBOMKRO.sqlite" --passphrase="MY_PASSPHRASE" --full-remote-verification=true --full-block-verification=true --full-result'

This results in >5000 lines/errors with “Extra”.

[...]
duplicati-id1c08b91f424414581362d679cb6f0c8.dindex.zip.aes: 2 errors
	Extra: oKd+siVp4tPx9YjPWBuGiW4COmoK0Nwpc0lMgaHsGJk=
	Extra: sMLWKhff7Z41PVAQAiFn6FTfFQ5mxD3u/hPGm4RTojo=
duplicati-i55f6fce9390a44a8b748831269181095.dindex.zip.aes: No errors
duplicati-i425b340a60574da3b3e00d7709d64301.dindex.zip.aes: No errors
duplicati-ie9ca759663894befa58f2246dce72053.dindex.zip.aes: 556 errors
	Extra: +/6EatqrbOiaKB+ZnNWE4/CoIwB8XKxQaCa5v3oDCXk=	
	Extra: /6nsFrzXP4SgsXQWElf6XejFT3mbcwzDXksOg4bbO4o=
	Extra: /F0GDl8vVjyUEt0kY0pz02TVBkVKXpturGMJ0rlIYHE=
	Extra: /INZFP4NgLWUY7N2L8LF0ePAEArW5krwi/d4cKhUiAk=
	Extra: /LzjZNw71HGoN8J0N+Mv7lfxxJuxk/0Juxas52881wU=
	Extra: /NhAxJYdgXXHcYxwsaORtgB5iowqHbl+8hjbhF+OfN4=
	Extra: 00ijmg3We/g95IvdayQaU8nO1UuhydKDdut75JpC5H4=
	Extra: 06qJ44QGl+vdJ3gqxH0FHFwKcPWNABJKKwmBI/THKwU=
	Extra: 09MJuyNi5pWfxOhIZcKy6PXRmpQ+ZN+35n4EvLn3mxo=
	Extra: 0Du5LX0LgcDJ/StcrFmcSA59L/60u2170l4131f6sdI=
	Extra: 0N0ksS+h5AodLiHQe5ww6UzlBNKUtvOnowmySqImSSc=
[...]

full log is here : https://pastes.io/ujlqbw3qe8
What does this mean?

@ts678

Is this first such test? If so, please look at the dates of offending files to see how old they are.
Older releases were more likely to be a little off, although I think this look can still be induced.

Sometimes the leftover dindex file is needless, but showing that is difficult without some work.

Here are the dates from the affected dindex files:

duplicati-i35985736e69e4faea5a5bc8ce7b6ea6f.dindex.zip.aes	18.4 KB	01/15/2024 18:07
duplicati-id48eceb4bdd44bf9a28901bdd165ed95.dindex.zip.aes	18.4 KB	07/08/2023 00:46
duplicati-ie9ca759663894befa58f2246dce72053.dindex.zip.aes	27.6 KB	05/30/2022 14:51

So the oldest affected dindex is from May 2022. I think I set up the backup job beginning of 2022 with a new PC. Do you consider this an old release?

Btw, I also tried restoring the files form the latest snapshot and this was working without problems. Having this many errors in the database is still not very reassuring

[ts678]

@babelfish

What does this mean?

Good news. Extra is harmless duplication of blocks that ideally would only occur once. None missing.
Fix “test” error: extra hashes #4982 may go in someday, but it’s best to be cautious with core change.

@babelfish

affected dindex files

These are likely extra left-behind dindex files. The dblocks they mention aren’t found in the download. Each dindex will make lots of “Extra” complaints because it cites blocks that aren’t now in the backup.

One way to show that they’re useless is to rename them with a different prefix, and do recreate again.
Watching About → Show log → Live → Verbose should ideally show only dlist and dindex downloads.

Watching the progress bar, this should go to 70%. Anything beyond that is looking into the big dblocks because the dindex files did not have enough info to know which dblock all the needed blocks were in.

[babelfish]

I deleted the 3 dindex files (and the db) and now recreate is running without any errors/warnings

Are the “Extra” lines blocks which will be compacted once there are enough in a dblock file?

[ts678]

@babelfish

recreate is running without any errors/warnings

I guess that’s so-far-so-good, but keep an eye on progress. Ideally, there are no dblock reads.

@babelfish

Are the “Extra” lines blocks which will be compacted once there are enough in a dblock file?

I doubt it. The database has just one dblock listed for a block. Extras exist outside of records…

[babelfish]

hm, so what does this mean now in general? Is this “normal”? Or are there some undiscovered bugs causing this?

[ts678]

Please read the previously provided link for a summary of the fix, and a pointer to the discovered bug:

@ts678

Fix “test” error: extra hashes #4982 may go in someday, but it’s best to be cautious with core change.

EDIT:

The way to get undiscovered bugs is to slam in a fix for a harmless problem. The cure might be worse.

[ts678]

Ordinarily there are just a few Extra per file. Only small fraction of a percent of storage is wasted.
This part is normal (until it’s decided that the attempted fix is worth it), and interferes with nothing.

You had a second problem, which actually led to you discovering the harmless Extra. Proposal is:

Fix leftover/extra index volume after interrupted upload and more tests

[babelfish]

Thanks for the explanation!

@ts678

The way to get undiscovered bugs is to slam in a fix for a harmless problem. The cure might be worse.

I agree, defnitly don’t rush changes/fixes on the DB