Random files not found on Backblaze target

Support
1

My backup on Backblaze started to show errors 3 days after updating to 2.1.0.4_stable_2025-01-31.
Update on the 31st, 1st and 2nd went without errors. On 3rd and 4th of February there were “File … not found” errors, but the backups “succeeded” . After that the backups failed until today when I noticed the issue.
Database repairs are successful and only take seconds. But a Backup or Test that fail again. But not always. I managed to run a repair and a subsequent backup and that time it went through without any errors.

I ran a couple of archive tests and noticed that it always shows these errors:
[Error-Duplicati.Library.Main.Operation.TestHandler-RemoteFileProcessingFailed]: Failed to process file …
FileMissingException: File … not found

BUT the the files are seemingly random. Subsequent test runs show different files missing.

I then checked my bucket on Backblaze and those files that are reported missing, exist and can be downloaded.

Could it be that the latest backup broke the Backblaze S3 integration?
Never had this error in the years before.

2

FWIW it works for me, but I did only a small backup and ran BackendTool and BackendTester.
Feed those a target URL similar to what Export As Command-line gives you.

Someone else is having mysterious S3 problems, and you can see the methods and talk here. Running some of those tests can see if your issue is similar. It’s possible these need dev help.

The TEST command

Verifies integrity of a backup. A random sample of dlist, dindex, dblock files is downloaded, decrypted and the content is checked against recorded size values and data hashes.

so even though the test sample is somewhat random, is everything selected failing that way?

3

Yesterday a backup was successful without any errors.
Today Backup failed and reported 6 extra unknown files and asked for repair.
I ran repair and it reported that the 6 files were deleted. At least that’s hwo I understand it:

    "2025-02-11 17:36:07 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: Die Operation Repair wurde gestartet",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (330 Bytes)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20250104T165834Z.dlist.zip.aes (4,56 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20250104T165834Z.dlist.zip.aes (4,56 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20250127T164514Z.dlist.zip.aes (4,73 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20250127T164514Z.dlist.zip.aes (4,73 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20250128T164519Z.dlist.zip.aes (4,73 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20250128T164519Z.dlist.zip.aes (4,73 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20250130T163006Z.dlist.zip.aes (4,66 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20250130T163006Z.dlist.zip.aes (4,66 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20250131T163006Z.dlist.zip.aes (4,67 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20250131T163006Z.dlist.zip.aes (4,67 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20250202T163007Z.dlist.zip.aes (4,68 MB)",
    "2025-02-11 17:36:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20250202T163007Z.dlist.zip.aes (4,68 MB)"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 7,
    "BytesUploaded": 0,
    "BytesDownloaded": 0,
    "FilesUploaded": 0,
    "FilesDownloaded": 0,
    "FilesDeleted": 6,

I then ran a manual backup and it failed because 2 files were not found.

I checked the content of the bucket with BackendTool and the files that are supposedly missing and the ones that should’ve been deleted are all still in the bucket.
I actually ran LIST 10 times and the result is always the same.

All weird or I misunderstand the messages.
But Backups are randomly working and failing.

4

Are these from TestHandler like in original post? How many files does that get. e.g. in log

image
image258×116 5.23 KB

and the Complete log names them (although it might run out of room). Mine all work here

    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20250211T142146Z.dlist.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-i4082e60afa8c49b78b45b86796b9bf21.dindex.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-be71f08397e1f426f9e6ea20b04b2da2a.dblock.zip.aes",
        "Value": []
      }
    ],

Since you’ve done this sort of looking a few times, did you notice if the files are old or new?
If you have file names, the dlist file actually has its own date/time as part of its file name.

Another test you can do is Verify files, which is like the test after backup but likely later.

Does Restore work? That also requires file downloads.

5

No, after that

grafik
grafik1663×580 67.5 KB

            {
  "DeletedFiles": 217,
  "DeletedFolders": 11,
  "ModifiedFiles": 464,
  "ExaminedFiles": 41351,
  "OpenedFiles": 756,
  "AddedFiles": 292,
  "SizeOfModifiedFiles": 313711664,
  "SizeOfAddedFiles": 163182063,
  "SizeOfExaminedFiles": 16105718851,
  "SizeOfOpenedFiles": 499957682,
  "NotProcessedFiles": 0,
  "AddedFolders": 30,
  "TooLargeFiles": 0,
  "FilesWithError": 0,
  "ModifiedFolders": 0,
  "ModifiedSymlinks": 0,
  "AddedSymlinks": 0,
  "DeletedSymlinks": 0,
  "PartialBackup": false,
  "Dryrun": false,
  "MainOperation": "Backup",
  "CompactResults": {
    "DeletedFileCount": 0,
    "DownloadedFileCount": 0,
    "UploadedFileCount": 0,
    "DeletedFileSize": 0,
    "DownloadedFileSize": 0,
    "UploadedFileSize": 0,
    "Dryrun": false,
    "VacuumResults": null,
    "MainOperation": "Compact",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
    "EndTime": "2025-02-11T16:38:14.1236457Z",
    "BeginTime": "2025-02-11T16:38:09.7304784Z",
    "Duration": "00:00:04.3931673",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 17,
      "BytesUploaded": 125589575,
      "BytesDownloaded": 4310221,
      "FilesUploaded": 3,
      "FilesDownloaded": 1,
      "FilesDeleted": 1,
      "FoldersCreated": 0,
      "RetryAttempts": 8,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 332,
      "KnownFileSize": 60809544652,
      "LastBackupDate": "2025-02-11T17:36:51+01:00",
      "BackupListCount": 21,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2025-02-11T16:36:51.4764191Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "VacuumResults": null,
  "DeleteResults": {
    "DeletedSetsActualLength": 1,
    "DeletedSets": [
      {
        "Item1": 4,
        "Item2": "2025-02-03T17:52:31+01:00"
      }
    ],
    "Dryrun": false,
    "MainOperation": "Delete",
    "CompactResults": {
      "DeletedFileCount": 0,
      "DownloadedFileCount": 0,
      "UploadedFileCount": 0,
      "DeletedFileSize": 0,
      "DownloadedFileSize": 0,
      "UploadedFileSize": 0,
      "Dryrun": false,
      "VacuumResults": null,
      "MainOperation": "Compact",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
      "EndTime": "2025-02-11T16:38:14.1236457Z",
      "BeginTime": "2025-02-11T16:38:09.7304784Z",
      "Duration": "00:00:04.3931673",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null,
      "BackendStatistics": {
        "RemoteCalls": 17,
        "BytesUploaded": 125589575,
        "BytesDownloaded": 4310221,
        "FilesUploaded": 3,
        "FilesDownloaded": 1,
        "FilesDeleted": 1,
        "FoldersCreated": 0,
        "RetryAttempts": 8,
        "UnknownFileSize": 0,
        "UnknownFileCount": 0,
        "KnownFileCount": 332,
        "KnownFileSize": 60809544652,
        "LastBackupDate": "2025-02-11T17:36:51+01:00",
        "BackupListCount": 21,
        "TotalQuotaSpace": 0,
        "FreeQuotaSpace": 0,
        "AssignedQuotaSpace": -1,
        "ReportedQuotaError": false,
        "ReportedQuotaWarning": false,
        "MainOperation": "Backup",
        "ParsedResult": "Success",
        "Interrupted": false,
        "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
        "EndTime": "0001-01-01T00:00:00",
        "BeginTime": "2025-02-11T16:36:51.4764191Z",
        "Duration": "00:00:00",
        "MessagesActualLength": 0,
        "WarningsActualLength": 0,
        "ErrorsActualLength": 0,
        "Messages": null,
        "Warnings": null,
        "Errors": null
      }
    },
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
    "EndTime": "2025-02-11T16:38:14.1236483Z",
    "BeginTime": "2025-02-11T16:38:01.2676406Z",
    "Duration": "00:00:12.8560077",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 17,
      "BytesUploaded": 125589575,
      "BytesDownloaded": 4310221,
      "FilesUploaded": 3,
      "FilesDownloaded": 1,
      "FilesDeleted": 1,
      "FoldersCreated": 0,
      "RetryAttempts": 8,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 332,
      "KnownFileSize": 60809544652,
      "LastBackupDate": "2025-02-11T17:36:51+01:00",
      "BackupListCount": 21,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2025-02-11T16:36:51.4764191Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "RepairResults": null,
  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20250211T163651Z.dlist.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-i43c1c42143aa40d99c3634d88607cf35.dindex.zip.aes",
        "Value": [
          {
            "Key": "Error",
            "Value": "File duplicati-i43c1c42143aa40d99c3634d88607cf35.dindex.zip.aes not found"
          }
        ]
      },
      {
        "Key": "duplicati-b1fad228e3b95436596582357f6353918.dblock.zip.aes",
        "Value": [
          {
            "Key": "Error",
            "Value": "File duplicati-b1fad228e3b95436596582357f6353918.dblock.zip.aes not found"
          }
        ]
      }
    ],
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
    "EndTime": "2025-02-11T16:39:37.0523945Z",
    "BeginTime": "2025-02-11T16:38:14.260745Z",
    "Duration": "00:01:22.7916495",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 17,
      "BytesUploaded": 125589575,
      "BytesDownloaded": 4310221,
      "FilesUploaded": 3,
      "FilesDownloaded": 1,
      "FilesDeleted": 1,
      "FoldersCreated": 0,
      "RetryAttempts": 8,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 332,
      "KnownFileSize": 60809544652,
      "LastBackupDate": "2025-02-11T17:36:51+01:00",
      "BackupListCount": 21,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2025-02-11T16:36:51.4764191Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Error",
  "Interrupted": false,
  "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
  "EndTime": "2025-02-11T16:39:37.0586137Z",
  "BeginTime": "2025-02-11T16:36:51.4764165Z",
  "Duration": "00:02:45.5821972",
  "MessagesActualLength": 55,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 3,
  "Messages": [
    "2025-02-11 17:36:51 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: Die Operation Backup wurde gestartet",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (330 Bytes)",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-IgnoreRemoteDeletedFile]: ignoring remote file listed as Deleted: duplicati-20250104T165834Z.dlist.zip.aes",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-IgnoreRemoteDeletedFile]: ignoring remote file listed as Deleted: duplicati-20250127T164514Z.dlist.zip.aes",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-IgnoreRemoteDeletedFile]: ignoring remote file listed as Deleted: duplicati-20250128T164519Z.dlist.zip.aes",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-IgnoreRemoteDeletedFile]: ignoring remote file listed as Deleted: duplicati-20250130T163006Z.dlist.zip.aes",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-IgnoreRemoteDeletedFile]: ignoring remote file listed as Deleted: duplicati-20250131T163006Z.dlist.zip.aes",
    "2025-02-11 17:36:57 +01 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-IgnoreRemoteDeletedFile]: ignoring remote file listed as Deleted: duplicati-20250202T163007Z.dlist.zip.aes",
    "2025-02-11 17:37:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bf2c5da9a18334263ad715af5c1c3f828.dblock.zip.aes (115,43 MB)",
    "2025-02-11 17:38:00 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bf2c5da9a18334263ad715af5c1c3f828.dblock.zip.aes (115,43 MB)",
    "2025-02-11 17:38:00 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-id911ae551a0e47a6a3684a162c937e6c.dindex.zip.aes (239,33 KB)",
    "2025-02-11 17:38:00 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-id911ae551a0e47a6a3684a162c937e6c.dindex.zip.aes (239,33 KB)",
    "2025-02-11 17:38:00 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20250211T163651Z.dlist.zip.aes (4,11 MB)",
    "2025-02-11 17:38:01 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20250211T163651Z.dlist.zip.aes (4,11 MB)",
    "2025-02-11 17:38:01 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed",
    "2025-02-11 17:38:01 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 31.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00, Unlimited / 365.00:00:00",
    "2025-02-11 17:38:01 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 10.02.2025 17:30:02, 09.02.2025 17:30:01, 08.02.2025 14:05:30, 03.02.2025 17:52:31, 01.02.2025 19:55:22, 25.01.2025 19:03:36, 18.01.2025 18:59:32, 11.01.2025 18:36:01, 27.12.2024 10:10:58, 24.11.2024 17:30:06, 22.10.2024 18:30:06, 15.09.2024 09:29:33, 08.08.2024 18:30:06, 02.07.2024 18:30:06, 01.06.2024 18:30:06, 01.05.2024 18:30:06, 25.03.2024 17:45:39, 22.02.2024 17:34:29, 29.09.2023 21:58:38, 03.09.2022 09:27:54, 05.08.2021 17:17:02",
    "2025-02-11 17:38:01 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted: ",
    "2025-02-11 17:38:01 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 03.02.2025 17:52:31"
  ],
  "Warnings": [],
  "Errors": [
    "2025-02-11 17:38:55 +01 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-i43c1c42143aa40d99c3634d88607cf35.dindex.zip.aes\r\nFileMissingException: File duplicati-i43c1c42143aa40d99c3634d88607cf35.dindex.zip.aes not found",
    "2025-02-11 17:39:37 +01 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b1fad228e3b95436596582357f6353918.dblock.zip.aes\r\nFileMissingException: File duplicati-b1fad228e3b95436596582357f6353918.dblock.zip.aes not found",
    "2025-02-11 17:39:37 +01 - [Error-Duplicati.Library.Main.Operation.TestHandler-Test results]: Verified 3 remote files with 2 problem(s)"
  ],
  "BackendStatistics": {
    "RemoteCalls": 17,
    "BytesUploaded": 125589575,
    "BytesDownloaded": 4310221,
    "FilesUploaded": 3,
    "FilesDownloaded": 1,
    "FilesDeleted": 1,
    "FoldersCreated": 0,
    "RetryAttempts": 8,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 332,
    "KnownFileSize": 60809544652,
    "LastBackupDate": "2025-02-11T17:36:51+01:00",
    "BackupListCount": 21,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.1.0.4 (2.1.0.4_stable_2025-01-31)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2025-02-11T16:36:51.4764191Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

Not sure what you mean. The files were all the same. I checked the 10 outputs for differences and they are exactly the same.

Name	Dir/File	LastChange	Size
duplicati-20210805T151702Z.dlist.zip.aes	File	05.08.2021 17:40:19	1,43 MB
duplicati-20220903T072754Z.dlist.zip.aes	File	03.09.2022 09:38:25	11,62 MB
duplicati-20230929T195838Z.dlist.zip.aes	File	29.09.2023 22:02:51	2,59 MB
duplicati-20240222T163429Z.dlist.zip.aes	File	22.02.2024 17:38:18	2,82 MB
duplicati-20240325T164539Z.dlist.zip.aes	File	25.03.2024 17:50:30	2,88 MB
duplicati-20240501T163006Z.dlist.zip.aes	File	01.05.2024 18:36:32	2,90 MB
duplicati-20240601T163006Z.dlist.zip.aes	File	01.06.2024 18:34:12	2,91 MB
duplicati-20240702T163006Z.dlist.zip.aes	File	02.07.2024 18:34:26	2,97 MB
duplicati-20240808T163006Z.dlist.zip.aes	File	08.08.2024 18:37:38	5,04 MB
duplicati-20240915T072933Z.dlist.zip.aes	File	15.09.2024 09:38:14	5,65 MB
duplicati-20241022T163006Z.dlist.zip.aes	File	22.10.2024 18:34:43	3,65 MB
duplicati-20241124T163006Z.dlist.zip.aes	File	24.11.2024 17:36:17	4,26 MB
duplicati-20241227T091058Z.dlist.zip.aes	File	27.12.2024 10:18:23	4,55 MB
duplicati-20250104T165834Z.dlist.zip.aes	File	04.01.2025 18:05:52	4,56 MB
duplicati-20250111T173601Z.dlist.zip.aes	File	11.01.2025 18:43:26	4,61 MB
duplicati-20250118T175932Z.dlist.zip.aes	File	18.01.2025 19:07:18	4,66 MB
duplicati-20250125T180336Z.dlist.zip.aes	File	25.01.2025 19:11:16	4,70 MB
duplicati-20250127T164514Z.dlist.zip.aes	File	27.01.2025 17:53:04	4,73 MB
duplicati-20250128T164519Z.dlist.zip.aes	File	28.01.2025 17:53:09	4,73 MB
duplicati-20250130T163006Z.dlist.zip.aes	File	30.01.2025 17:37:40	4,66 MB
duplicati-20250131T163006Z.dlist.zip.aes	File	31.01.2025 17:37:40	4,67 MB
duplicati-20250201T185522Z.dlist.zip.aes	File	01.02.2025 20:03:04	4,67 MB
duplicati-20250202T163007Z.dlist.zip.aes	File	02.02.2025 17:38:07	4,68 MB
duplicati-b0029516a9ba8444ab1ba72ce90dab2fb.dblock.zip.aes	File	24.01.2025 17:37:18	101,10 MB
duplicati-b06f55a4e87314e06b116e217129a383e.dblock.zip.aes	File	12.11.2021 20:26:04	511,98 MB
duplicati-b075d2bfc32da4070a05f970eca84f1c2.dblock.zip.aes	File	27.07.2023 18:35:21	512,02 MB
duplicati-b0a19e77f3d3a4421a790da6c6c3d8cf1.dblock.zip.aes	File	05.08.2021 17:20:03	511,93 MB
duplicati-b0a288fe7b2664cc681c14430e5aacdd0.dblock.zip.aes	File	19.09.2023 18:34:13	250,52 MB
duplicati-b0b2baafa7fe24816a4a9f546a309f921.dblock.zip.aes	File	03.06.2024 18:36:40	512,00 MB
duplicati-b143aaa6dcf36470d8ed88567676bab2b.dblock.zip.aes	File	31.01.2025 17:37:17	108,13 MB
duplicati-b14edddf1666b447c9e2331581a2cb40e.dblock.zip.aes	File	12.11.2021 19:19:59	511,96 MB
duplicati-b18189060dc9a4add8fb1cef1240ecd6c.dblock.zip.aes	File	12.11.2021 19:50:20	511,98 MB
duplicati-b1895abbad22c43afaf466a3cdd9d6cc1.dblock.zip.aes	File	12.11.2021 19:06:32	511,92 MB
duplicati-b1c056bb9e2d24724863f55a3c2530928.dblock.zip.aes	File	02.01.2025 18:11:14	511,99 MB
duplicati-b1c63aa49df1e4391bf81ff4f8dbddbdb.dblock.zip.aes	File	12.11.2021 19:31:21	511,98 MB
duplicati-b1c75db4212ca4c8a982731080b5dba4d.dblock.zip.aes	File	25.07.2024 18:49:45	512,00 MB
duplicati-b1d4b71c40bbe4269b04d0e690fd28c86.dblock.zip.aes	File	25.03.2024 17:50:03	132,56 MB
duplicati-b1de3af5ea5fb49188205e86b70c7bf6a.dblock.zip.aes	File	05.08.2021 17:19:20	511,91 MB
duplicati-b1ed002b9cab244bdab612c2cec47013a.dblock.zip.aes	File	20.10.2024 08:06:16	112,93 MB
duplicati-b1fad228e3b95436596582357f6353918.dblock.zip.aes	File	28.01.2025 17:52:43	110,20 MB
duplicati-b23481645f4e14561b80f610bfdff9208.dblock.zip.aes	File	27.12.2024 10:17:55	128,06 MB
duplicati-b25cb33b4e15a462f9f14c7836adf9a20.dblock.zip.aes	File	12.11.2021 20:42:32	511,93 MB
duplicati-b279f164ae9b84d75861582565cb861d6.dblock.zip.aes	File	05.08.2021 17:19:49	511,95 MB
duplicati-b2c5023b2bd9e4397a6e1161391f78e81.dblock.zip.aes	File	12.11.2021 20:08:49	511,91 MB
duplicati-b2fa17414e8fc47e6852d9bb9ad317417.dblock.zip.aes	File	05.08.2021 17:32:05	511,94 MB
duplicati-b335f93f22420403b8c65c05085f7e08b.dblock.zip.aes	File	17.01.2025 17:50:25	512,00 MB
duplicati-b355fe80bfa2c43b7a562240243a8f3e4.dblock.zip.aes	File	05.08.2024 18:38:43	512,06 MB
duplicati-b364aa25fd93e44b98f2559f7b9023c00.dblock.zip.aes	File	12.11.2021 20:17:36	511,99 MB
duplicati-b397bad66b90745309d6d93f362655d5a.dblock.zip.aes	File	17.01.2025 17:58:06	432,52 MB
duplicati-b39dc3dc4f3e1493b847429e11246b909.dblock.zip.aes	File	12.11.2021 19:02:43	511,91 MB
duplicati-b3b321d8db8e342e3b675d2ba550d0aaa.dblock.zip.aes	File	03.03.2024 17:37:55	512,02 MB
duplicati-b3bd4980eaed34143bb5f9aedcc6ac877.dblock.zip.aes	File	03.08.2022 23:55:06	211,17 MB
duplicati-b3ddeee0ae0944c45bd857643e25790ac.dblock.zip.aes	File	11.01.2025 18:43:00	120,53 MB
duplicati-b41342eb89f44411aa656dce04608ee27.dblock.zip.aes	File	12.11.2021 20:44:01	511,96 MB
duplicati-b42ab0504dd5643ee83d7cf41b767b8b1.dblock.zip.aes	File	04.11.2024 19:13:56	512,04 MB
duplicati-b42c0efec024f4d39ad53daa256cebd15.dblock.zip.aes	File	12.11.2021 19:40:16	511,96 MB
duplicati-b43ff4daa1b1d447291d5a03fe7d63caa.dblock.zip.aes	File	23.06.2023 18:49:08	512,02 MB
duplicati-b492247accc0445eaafe86c3eee17db2e.dblock.zip.aes	File	12.11.2021 19:52:07	511,99 MB
duplicati-b4a95dbfe5e154b66ab364b95caa552aa.dblock.zip.aes	File	20.06.2024 18:34:51	511,96 MB
duplicati-b4bbd9e5a51e04bfdb5a2264f075c0c19.dblock.zip.aes	File	22.01.2025 17:41:02	105,06 MB
duplicati-b4c951eade1524e048227525d77b4b146.dblock.zip.aes	File	12.11.2021 20:37:47	511,97 MB
duplicati-b4e4bec48b5644f59b1a4408117664655.dblock.zip.aes	File	12.11.2021 20:11:55	511,96 MB
duplicati-b4efeb302f3894cc2bd0c14dfc3643761.dblock.zip.aes	File	12.11.2021 19:27:33	511,98 MB
duplicati-b52b1eb2c06244b26b6a0cf290576bf33.dblock.zip.aes	File	17.01.2025 17:42:21	512,01 MB
duplicati-b5410dedac26f4754aceb88e9c3e84d2a.dblock.zip.aes	File	12.11.2021 20:22:36	511,95 MB
duplicati-b5606b6d9be9945f2b6f3f7f25d03de2e.dblock.zip.aes	File	29.01.2025 17:37:33	128,50 MB
duplicati-b5643dc8eb28443ad806f2ad68c1e4d42.dblock.zip.aes	File	17.04.2024 18:43:26	512,04 MB
duplicati-b57d285e23cc84ee39c063200f22c201d.dblock.zip.aes	File	12.11.2021 19:29:02	511,96 MB
duplicati-b5968ea0aa8a348dba22d0afe71412bc0.dblock.zip.aes	File	17.11.2024 09:29:55	512,05 MB
duplicati-b5c401282148a4cc386e399270bbc79ad.dblock.zip.aes	File	21.01.2025 17:58:36	84,86 MB
duplicati-b5cf1f54147794f94a54bff1ee0337f5e.dblock.zip.aes	File	12.11.2021 19:17:10	512,00 MB
duplicati-b5cfa1c1bfc2949b085ced8f1049118c6.dblock.zip.aes	File	15.01.2025 18:00:40	106,07 MB
duplicati-b5db82fe01f9c495da0ddd37878ecaf37.dblock.zip.aes	File	12.11.2021 19:49:43	511,93 MB
duplicati-b6137e5bf526a4b4a9239b5d53e9e4654.dblock.zip.aes	File	28.06.2024 18:35:00	511,97 MB
duplicati-b616f9304bbc4429da9bb9cbf4a2fc550.dblock.zip.aes	File	14.12.2024 17:37:48	512,04 MB
duplicati-b6182553b626c4857a051c1ca8efa4032.dblock.zip.aes	File	19.03.2024 17:35:51	512,09 MB
duplicati-b63045acad3e44b98a7db337a4fc3aa28.dblock.zip.aes	File	12.11.2021 19:08:18	511,90 MB
duplicati-b6622d34fffea47d7966167587c949326.dblock.zip.aes	File	12.11.2021 20:33:22	511,96 MB
duplicati-b6661eacddd414ad687e5c097e01ab88c.dblock.zip.aes	File	19.01.2023 17:57:49	387,05 MB
duplicati-b68c7df4342cd43378b6ecf174b9c0df9.dblock.zip.aes	File	04.11.2024 19:11:26	512,07 MB
duplicati-b6b3035de0d61416388f1d1462f489cce.dblock.zip.aes	File	12.11.2021 18:55:47	511,95 MB
duplicati-b6de58965d46642ef9886449d17bccf58.dblock.zip.aes	File	08.10.2024 18:41:38	512,08 MB
duplicati-b6e1d442222f64720aa940c7a0c024827.dblock.zip.aes	File	18.01.2025 19:06:50	132,98 MB
duplicati-b6e74f56b7b2f44de88859b96c47c5175.dblock.zip.aes	File	03.10.2024 18:32:48	416,83 MB
duplicati-b6e8fa046746846b78c908ea6c08219d0.dblock.zip.aes	File	08.10.2024 18:44:14	512,08 MB
duplicati-b6f11fce8dac340ed9e15bd371082f0a8.dblock.zip.aes	File	25.07.2024 18:52:11	512,07 MB
duplicati-b6fa0045abb39409d973a86bb7826cd42.dblock.zip.aes	File	04.10.2024 18:33:58	511,92 MB
duplicati-b74c4cada63b241caa725b1ec62be1ff6.dblock.zip.aes	File	14.07.2024 09:36:54	160,66 MB
duplicati-b74e0c4a7d1614a71b49b3f9118556112.dblock.zip.aes	File	25.08.2024 08:52:39	512,03 MB
duplicati-b7508c1843b9148e4a0a65c2c7b998783.dblock.zip.aes	File	03.08.2022 23:54:52	511,95 MB
duplicati-b75d872e80f024736984521ecce46eb81.dblock.zip.aes	File	04.01.2025 18:05:29	110,23 MB
duplicati-b77ba02a94e6e4851954b50c73f7a3779.dblock.zip.aes	File	02.02.2025 17:37:39	113,79 MB
duplicati-b793b5140372248d0a1edd6e371b2b882.dblock.zip.aes	File	12.11.2021 18:45:26	511,95 MB
duplicati-b79e1f71005a6402b961cb152e5a9469e.dblock.zip.aes	File	12.11.2021 18:53:00	511,94 MB
duplicati-b7a6fcd013a0a41bd94d900cac782168f.dblock.zip.aes	File	30.01.2025 17:37:16	106,30 MB
duplicati-b7fcf4219fe5f4a689c3187b8e092f395.dblock.zip.aes	File	12.11.2021 20:32:22	511,99 MB
duplicati-b82e0dcfe2d914381afabe129f2afb66d.dblock.zip.aes	File	20.01.2025 17:37:26	93,00 MB
duplicati-b85da3850c06a4bbc8ed782fa305939d0.dblock.zip.aes	File	03.03.2024 17:46:16	512,01 MB
duplicati-b8681f2cee0d940a49d5b74b5c824ca3e.dblock.zip.aes	File	02.01.2025 18:11:24	203,32 MB
duplicati-b876183a83ef74b638e66759208d91e18.dblock.zip.aes	File	17.01.2025 17:44:53	512,00 MB
duplicati-b8a62f0659ddd491c8e0017e4d37f1f1a.dblock.zip.aes	File	12.11.2021 19:21:29	511,99 MB
duplicati-b8b20b7bc0fda430f8333dcdef51abd98.dblock.zip.aes	File	31.05.2024 18:34:09	111,33 MB
duplicati-b8c06593f0a4240eabe66b75c92a2a6a3.dblock.zip.aes	File	25.08.2024 08:57:54	512,03 MB
duplicati-b8ebd56962d704821bb291b4af863fe9b.dblock.zip.aes	File	12.11.2021 20:46:27	511,92 MB
duplicati-b8ed203c80914430bb20e88d89dd7fe37.dblock.zip.aes	File	04.10.2024 18:33:39	511,92 MB
duplicati-b8fa282d054c44b8d9fdcedb90fe102bf.dblock.zip.aes	File	12.11.2021 18:34:01	511,98 MB
duplicati-b94fbd397e57a48e2b7d189aa69ad9e7e.dblock.zip.aes	File	12.11.2021 20:22:07	511,96 MB
duplicati-b95eaec1102714e4aab4317f21dac80d5.dblock.zip.aes	File	12.11.2021 18:43:52	511,98 MB
duplicati-b9ae98bd2b26f460b875ea6547d4afdfe.dblock.zip.aes	File	19.01.2025 17:37:21	114,46 MB
duplicati-b9c590f08ffa4499ebcf97d668566a3f7.dblock.zip.aes	File	25.01.2025 19:10:51	111,69 MB
duplicati-b9cb81e5122ad4773904a839a692d7e80.dblock.zip.aes	File	05.08.2021 17:19:35	511,95 MB
duplicati-b9e3b42e20fef4954b481efda27b9866d.dblock.zip.aes	File	17.01.2025 17:38:53	512,01 MB
duplicati-ba250d4297f85473e804b7aa063ad340f.dblock.zip.aes	File	13.08.2021 19:31:26	380,87 MB
duplicati-ba5dfe934b7394a2e8ae1eeb9a80c9a74.dblock.zip.aes	File	25.07.2024 18:38:35	512,00 MB
duplicati-ba9073e4f85a141c498cc16333c75a4ae.dblock.zip.aes	File	12.11.2021 19:40:07	512,00 MB
duplicati-baa4dce833cb94b48a4b0d263d43afb58.dblock.zip.aes	File	12.11.2021 20:36:27	511,93 MB
duplicati-bad092ccaae074bbba91cd4121a940a90.dblock.zip.aes	File	28.06.2024 18:35:06	136,99 MB
duplicati-bafaf2a2b8e9742b38a8c3e8448274567.dblock.zip.aes	File	12.11.2021 19:18:34	511,96 MB
duplicati-bb29a29e1288d43148b584df441d43086.dblock.zip.aes	File	12.11.2021 19:59:31	511,97 MB
duplicati-bb3b86b2154b14ef194facc2d204b73e1.dblock.zip.aes	File	17.01.2025 17:54:02	512,04 MB
duplicati-bb4a2028ee531457daaed5b4a62696297.dblock.zip.aes	File	08.10.2024 18:34:08	511,93 MB
duplicati-bb5092f8d95934144bc22cad2a6bcbca1.dblock.zip.aes	File	12.11.2021 19:11:50	511,95 MB
duplicati-bb59c12940fea49f0bb962627eab0c371.dblock.zip.aes	File	12.11.2021 18:45:18	511,98 MB
duplicati-bb89024ef9f6748e48795ce58b06c91b2.dblock.zip.aes	File	12.11.2021 18:34:01	511,91 MB
duplicati-bbaf5b3ddc25f4cf1837eb3943d752050.dblock.zip.aes	File	12.11.2021 18:57:14	511,95 MB
duplicati-bbb3d215d27da4942844739347f06cb6f.dblock.zip.aes	File	05.08.2021 17:32:55	479,97 MB
duplicati-bbccde676abaf47069ba5b0413e4a4e66.dblock.zip.aes	File	23.01.2025 17:37:29	108,09 MB
duplicati-bbe846a0618de4fa3a48236efa12528b9.dblock.zip.aes	File	12.11.2021 19:38:58	511,93 MB
duplicati-bbfd77e1f374f4e2398fa43c5877fc4d0.dblock.zip.aes	File	08.10.2024 18:34:45	511,99 MB
duplicati-bc16104cb60b642158bce8da86dd410d6.dblock.zip.aes	File	12.11.2021 20:11:03	511,96 MB
duplicati-bc2d789f8515c4257b5f5d9a9633e313e.dblock.zip.aes	File	27.01.2025 17:52:40	115,59 MB
duplicati-bc31ad03c1b1842f3aae7bc2950ece735.dblock.zip.aes	File	12.11.2021 19:50:08	511,94 MB
duplicati-bc6b48c41f7154b819d0f4b45f4a64d51.dblock.zip.aes	File	12.11.2021 19:41:22	511,99 MB
duplicati-bc7320113e6074642aea3cd38f6f5497c.dblock.zip.aes	File	26.01.2025 17:37:38	122,49 MB
duplicati-bc89f0762197e4964919b8eae0c86969c.dblock.zip.aes	File	12.11.2021 20:00:55	512,00 MB
duplicati-bcac56110eca348898d67b74c552930ef.dblock.zip.aes	File	07.10.2023 20:02:33	512,09 MB
duplicati-bcffda4e3bb134c5d92acaa6fb96876cc.dblock.zip.aes	File	12.11.2021 18:34:01	511,95 MB
duplicati-bd1730703a65e4ef38dc3857713528322.dblock.zip.aes	File	13.01.2025 18:24:33	104,23 MB
duplicati-bd2f1d07280b74741aa34570e9f918114.dblock.zip.aes	File	12.11.2021 20:14:30	512,00 MB
duplicati-bd3f7895607e446b490cfb8f7e9e3e645.dblock.zip.aes	File	12.11.2021 18:34:01	511,98 MB
duplicati-bd537575424c747baa753374d35582876.dblock.zip.aes	File	08.10.2024 18:34:27	511,92 MB
duplicati-bdaa9854e27144cbf9881324184508c61.dblock.zip.aes	File	07.05.2024 18:41:04	512,02 MB
duplicati-bdba6d3536a9e4df7a04659612b6d53ba.dblock.zip.aes	File	12.11.2021 20:27:15	511,93 MB
duplicati-bdca7977a18ed4e5aacaaefa947288c56.dblock.zip.aes	File	12.11.2021 18:57:38	511,93 MB
duplicati-bdd23c8056b534a5587b2aeb6c8c46969.dblock.zip.aes	File	12.11.2021 20:00:24	511,94 MB
duplicati-bde543fd63a0f4dfcb5f39e14a0db5fb2.dblock.zip.aes	File	04.10.2024 18:34:16	470,35 MB
duplicati-be279bde6140c43bbb8933c652f7d4a6c.dblock.zip.aes	File	12.11.2021 20:03:25	511,98 MB
duplicati-be6d397aa98994c438a5a42ed81a2ca86.dblock.zip.aes	File	01.02.2025 20:02:35	119,63 MB
duplicati-be6d9b675856e41cb85986baac9c67c4f.dblock.zip.aes	File	01.05.2024 18:34:34	489,14 MB
duplicati-beac982626e2b4ac3b4d265a599463d3e.dblock.zip.aes	File	16.06.2024 18:35:31	512,02 MB
duplicati-bed7e33bff203421f80651b7fba68d20e.dblock.zip.aes	File	12.11.2021 19:31:08	511,99 MB
duplicati-bf08f814b62cf4507b15587615a5344c8.dblock.zip.aes	File	12.11.2021 18:47:38	511,97 MB
duplicati-bf4723375710e4f30b62ef97ea6f75424.dblock.zip.aes	File	15.08.2022 20:16:40	192,67 MB
duplicati-bf550d2f941534f05aeabae85dd57c5a3.dblock.zip.aes	File	17.01.2025 17:47:23	512,09 MB
duplicati-bf77aea9c96944063beb1ee86789db3d1.dblock.zip.aes	File	25.08.2024 08:55:34	512,02 MB
duplicati-bfe64d26a93544050b7870bb851873843.dblock.zip.aes	File	12.11.2021 19:08:26	511,95 MB
...

Verify shows different errors on subsequent runs.
Logs are too big to be posted.

I doubt it works. I started it and see tons of errors in the live log. But maybe that’s a clue what is really wrong:

Duplicati.Library.Interface.FileMissingException: File duplicati-b74c4cada63b241caa725b1ec62be1ff6.dblock.zip.aes not found

 ---> Amazon.S3.AmazonS3Exception: Key not found
...

Key not found is strange.

6

means you said the missing files exist and can be downloaded. What were the dates of them?

Your listing shows all the file dates, except it seems to have been truncated, so lacks the files:

duplicati-i43c1c42143aa40d99c3634d88607cf35.dindex.zip.aes not found

duplicati-b1fad228e3b95436596582357f6353918.dblock.zip.aes not found

means

The file with dlist in its name has its date in its name too, so 20210805 matches 05.08.2021.

Log says Verified 3 remote files with 2 problem(s).The file that worked was new dlist:

duplicati-20250211T163651Z.dlist.zip.aes (and again 20250211 in name is backup’s date).

That is weakening the theory that maybe files were somehow not available right after an upload.

We don’t know how recent the two that failed are, until you look with your listing or B2’s web GUI.

That’s normal. It’s meant to be a sampling (with biases built in, such as testing never-tested files).
If a given file sometimes errors and sometimes doesn’t, that’s interesting. Different files is less so.

What part? Amazon S3 objects overview explains key. It’s like path. “not found” is current issue.

This raises the question – why did you choose S3? I wonder if Backblaze native runs any better?

Even if we keep chasing the Backblaze S3 issue, it’d be nice if Restore could run, if you needed it.

For S3 use, last I heard it was either required or recommended to use the right endpoint, seen on B2 Cloud Storage Buckets page where you can look at buckets and compare with Duplicati job.

Can you cause this problem on a small test job with only the essential options? Smaller is easier.

With what? It might be worth trying with BackendTool, with URL from Export As Command-line.

7

FWIW, this all to much of a hassle for me and I downgraded to 2.0.8.1_beta_2024-05-07.

First repair yielded a lot of errors different to the ones with the new version.

            {
  "MainOperation": "Repair",
  "RecreateDatabaseResults": {
    "MainOperation": "Repair",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)",
    "EndTime": "2025-02-11T21:09:21.6735164Z",
    "BeginTime": "2025-02-11T21:04:24.8860976Z",
    "Duration": "00:04:56.7874188",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 174,
      "BytesUploaded": 0,
      "BytesDownloaded": 196405977,
      "FilesUploaded": 0,
      "FilesDownloaded": 173,
      "FilesDeleted": 0,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 0,
      "KnownFileSize": 0,
      "LastBackupDate": "0001-01-01T00:00:00",
      "BackupListCount": 0,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": 0,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Repair",
      "ParsedResult": "Success",
      "Interrupted": false,
      "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2025-02-11T21:04:24.860246Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Error",
  "Interrupted": false,
  "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)",
  "EndTime": "2025-02-11T21:09:21.8773055Z",
  "BeginTime": "2025-02-11T21:04:24.860246Z",
  "Duration": "00:04:57.0170595",
  "MessagesActualLength": 353,
  "WarningsActualLength": 2,
  "ErrorsActualLength": 15,
  "Messages": [
    "2025-02-11 22:04:24 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: Die Operation Repair wurde gestartet",
    "2025-02-11 22:04:25 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2025-02-11 22:04:25 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (306 Bytes)",
    "2025-02-11 22:04:26 +01 - [Information-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-RebuildStarted]: Rebuild database started, downloading 23 filelists",
    "2025-02-11 22:04:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20210805T151702Z.dlist.zip.aes (1,43 MB)",
    "2025-02-11 22:04:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20210805T151702Z.dlist.zip.aes (1,43 MB)",
    "2025-02-11 22:04:26 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20220903T072754Z.dlist.zip.aes (11,62 MB)",
    "2025-02-11 22:04:27 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20220903T072754Z.dlist.zip.aes (11,62 MB)",
    "2025-02-11 22:04:28 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230929T195838Z.dlist.zip.aes (2,59 MB)",
    "2025-02-11 22:04:29 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230929T195838Z.dlist.zip.aes (2,59 MB)",
    "2025-02-11 22:04:48 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20240222T163429Z.dlist.zip.aes (2,82 MB)",
    "2025-02-11 22:04:48 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20240222T163429Z.dlist.zip.aes (2,82 MB)",
    "2025-02-11 22:04:52 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20240325T164539Z.dlist.zip.aes (2,88 MB)",
    "2025-02-11 22:04:53 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20240325T164539Z.dlist.zip.aes (2,88 MB)",
    "2025-02-11 22:04:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20240501T163006Z.dlist.zip.aes (2,90 MB)",
    "2025-02-11 22:04:57 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20240501T163006Z.dlist.zip.aes (2,90 MB)",
    "2025-02-11 22:05:00 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20240601T163006Z.dlist.zip.aes (2,91 MB)",
    "2025-02-11 22:05:01 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20240601T163006Z.dlist.zip.aes (2,91 MB)",
    "2025-02-11 22:05:04 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20240702T163006Z.dlist.zip.aes (2,97 MB)",
    "2025-02-11 22:05:04 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20240702T163006Z.dlist.zip.aes (2,97 MB)"
  ],
  "Warnings": [
    "2025-02-11 22:08:29 +01 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 15 missing volumes; attempting to replace blocks from existing volumes",
    "2025-02-11 22:09:01 +01 - [Warning-Duplicati.Library.Main.Database.LocalRecreateDatabase-MissingVolumesDetected]: Found 15 missing volumes; attempting to replace blocks from existing volumes"
  ],
  "Errors": [
    "2025-02-11 22:06:52 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bca29b9e75f4a464abcde060f80036940.dblock.zip.aes by duplicati-i2ee0c113476b4840a167dd05ebb09002.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:06:56 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bf13c8533954748ba90be4b66403a0c99.dblock.zip.aes by duplicati-i34b5d40d26dc413aa9c1f1b8303d6c00.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:06:59 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b286617dce8df481d880531563b34d1da.dblock.zip.aes by duplicati-i42b00a15e36c4b229363cdd73a00bb06.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:32 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bae322619a4194c0a912b88f810f04b6d.dblock.zip.aes by duplicati-i88d7586379014870bde147584b31278a.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:35 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bd165f63e731b4c1b8b583fca4b7d8b0d.dblock.zip.aes by duplicati-i8ff512b416a346aab0e4116281429431.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:41 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bc577c7eb5de74177890b14977f315ec6.dblock.zip.aes by duplicati-i9c1c49e50ab0438cafff12aecc33c7c8.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:48 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b276b9aaa13694f05ac0c51d704974ce8.dblock.zip.aes by duplicati-iac03158c42b64b05a837c55291f59069.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:48 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bbcff33eb2fce4e6283d65a1cd0a3edfd.dblock.zip.aes by duplicati-iadbe9390c88f4b7e814d5af088191969.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:51 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b5984930837cb46549cbe6679c403a296.dblock.zip.aes by duplicati-ib87f8e353abe4682b85fce76810901a8.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:53 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bf480bc7163f84d09a3d3214d96e7b992.dblock.zip.aes by duplicati-ib9f809113db44170afee47406de55ed4.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:07:55 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b53de82c3120f4a23970057df20d67370.dblock.zip.aes by duplicati-ibbcb34daaa454c98b48a18e31c3c018d.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:08:13 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b8ed930eed75245bebe01dc2e0c29cc35.dblock.zip.aes by duplicati-ie445ea4610c341c39e83a78357d433a6.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:08:18 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b2000b195311b4127ac33549db36e8772.dblock.zip.aes by duplicati-ie89b746c9f4e4d899b3ec72514905baf.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:08:25 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-bac293298bf024e02b75f47ed23e79cc7.dblock.zip.aes by duplicati-if49d7720e065403e8d7ae3e7f397d18e.dindex.zip.aes, but not found in list, registering a missing remote file",
    "2025-02-11 22:08:27 +01 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b58783df99c2a4018a61552e90f6cbb70.dblock.zip.aes by duplicati-ifadd10fd74844afd99773442fa9f5686.dindex.zip.aes, but not found in list, registering a missing remote file"
  ],
  "BackendStatistics": {
    "RemoteCalls": 174,
    "BytesUploaded": 0,
    "BytesDownloaded": 196405977,
    "FilesUploaded": 0,
    "FilesDownloaded": 173,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 0,
    "KnownFileSize": 0,
    "LastBackupDate": "0001-01-01T00:00:00",
    "BackupListCount": 0,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": 0,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Repair",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2025-02-11T21:04:24.860246Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

It told me to use “–rebuild-missing-dblock-files” which either didn’t work or I couldn’t make it work. Duplicati just seemed to be stuck and not doing anything.
I relaunched the program and recreated the DB which worked fine. Afterwards I did tests, repairs, a backup and 2 restores, one with the current backup one with a one year old version. Everything ran fine without errors.

Yes, because the post limit was reached. In all my tests the files that are supposed to me missing have all been present in the remote location.

The backup started years ago. I guess it was an option and just worked. I probably chose it in the first place because S3 is a widely used interface.

Backblaze web interface.

8

That and your other reports seems to clarify the issue is related to listing the B2 bucket having errors.

It is a bit troubling, because the S3 code was not changed from Duplicati’s side, and yet you report a downgrade fixes the issue.

@marceloduplicati Have you seen any cases in testing where a S3 file listing would omit files, seemingly at random?

9

No, I did not come across this.

There was an upgrade on AWS’s S3 client versions, but that would imply there is an issue with AWS’s latest version.

The other component is using S3 Client/API on Backblaze; on the B2 backend file listing operation uses B2 native API “b2api/v1/b2_list_file_versions” whereas if using the S3 backend, it will use S3 compatible apis.

My recent tests on B2 were focused around listing hundreds of thousands of files, but files created and uploaded with B2’s native API.

10

Totally understandable. Unfortunately, since we cannot reproduce the error locally, and you no longer experience the error, we cannot reliably fix it.

I was looking through the S3 code and there is no cache being done on lists. It is a direct call to the S3 API listing the bucket, so the only explanation I can think of is that the updated AWS library somehow has an issue when listing B2.

If you decide to upgrade again at some point, I would recommend switching to the native B2 backend.

11

Can I just switch the backend with the existing backup or do I have to start a new backup?

12

You can just edit the backup, change the connection info, and then it will work the same.

Duplicati only cares about the contents; it does not matter which backend connector you use.

13

FWIW that didn’t work.
I changed the connection settings and Test Connection did work, but then it ultimately complained that there are no remote files. The list command is just empty

Then I set up a fresh test backup and checked if that worked at all. It does.

Comparing old and new bucket I discovered an oddity with the files in the old bucket. All my files have a / at the beginning. In the folder path of the S3 backup settings is actually a single slash.
Doing the same in the B2 settings doesn’t help.

Well, my next plan is to download all files from Backblaze and upload them with fixed names to a new bucket.
That might finally fix the issues if the slash is the root of all those problems. (pun intended)

14

Like this? The discussion there might be useful, but I can’t really interpret what’s going on:

15

Nope.
Like
/duplicati-20240210T181146Z.dlist.zip.aes

And in the new bucket/backup they look like:

duplicati-20240210T181146Z.dlist.zip.aes

It even has weird consequences when downloading them from Backblaze. Single files will have the slash replaced with an underscore. Multiple are packed into a zip file which have the right size but show no files when being opened. Luckily extracting works and then the slash is stripped.

After work I will download my files and reupload them. Then try the backup again and hopefully upgrade.

16

That looks more like “Yep” because of the / at the beginning similar to the other confusing report. Probably the people who were working on that slash mystery (sounds like there was actual bug) could help with this one, if they think it applies. I’m just dropping a suggestion this topic is similar.

Forward slash is not legal in Windows file names, and is path separator on others such as Linux. Seems reasonable for Backblaze to do that. For zip oddity, results could vary depending on tool. Windows Explorer zip seems weaker than 7-Zip, and Linux zip tools might give the clearest view.

17

Final outcome:
After I downloaded all my files from Backblaze, I unpacked them and the slash at the start of each file name was, as expected, automatically removed by windows.
I uploaded all files to a new bucket and set it up with “B2 Cloud Storage” in Duplicati this time.
I rebuilt the database and was greeted with a new error.
15 x

[Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as ... by ..., but not found in list, registering a missing remote file"

As discussed here it is likely due to a failed backup in the past.
I did what the other user did and deleted all the these files. They were all from 2022, so not randomly across the years I used this backup.
First I did this in a local copy of the backup. After the files were removed (renaming while messing up the extensions does not work) no errors on repair, or test or verify or restore. (Minor warning due to a restored file being open during restore).
Did the same in the B2 bucket, without a full restore, also no errors.

Updated Duplicati, run a backup again, no errors.

So, now I’m back to a running backup with the current version.

1 Like
18

This issue has been “fixed” in 2.1.0.5, meaning it will not strip the leading slash anymore. So if you upgrade, beware that you need to modify the url for this to work again. We will probably add a check in the UI or similar that warns you if the paths start with a forward slash.

19

I already updated. The files have no slash anymore and I of course removed the slash from the folder setting of the connection.

1 Like