Restoring from Backblaze - 403 error - how should I interpret the logs?

Hi there,

I’ve been using Duplicati → Backblaze for years and never needed it until this past weekend when my external drive failed. I started a full data restore, it seems to have worked decently enough but in the logs I have these 32 errors:

2025-01-18 23:36:46 +00 - [Error-Duplicati.Library.Main.AsyncDownloader-FailedToRetrieveFile]: Failed to retrieve file duplicati-be5e200b026364c81835de527c26f22b2.dblock.zip
Exception: 403 - download_cap_exceeded: Cannot download file, download bandwidth or transaction (Class B) cap exceeded. See the Caps & Alerts page to increase your cap.
2025-01-18 23:36:46 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-PatchingFailed]: Failed to patch with remote file: "duplicati-be5e200b026364c81835de527c26f22b2.dblock.zip", message: 403 - download_cap_exceeded: Cannot download file, download bandwidth or transaction (Class B) cap exceeded. See the Caps & Alerts page to increase your cap.
Exception: 403 - download_cap_exceeded: Cannot download file, download bandwidth or transaction (Class B) cap exceeded. See the Caps & Alerts page to increase your cap.
2025-01-18 23:38:24 +00 - [Error-Duplicati.Library.Main.AsyncDownloader-FailedToRetrieveFile]: Failed to retrieve file duplicati-b5e542f0ae7f54484ac735f34887a0b9b.dblock.zip
Exception: 403 - download_cap_exceeded: Cannot download file, download bandwidth or transaction (Class B) cap exceeded. See the Caps & Alerts page to increase your cap.
2025-01-18 23:38:24 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-PatchingFailed]: Failed to patch with remote file: "duplicati-b5e542f0ae7f54484ac735f34887a0b9b.dblock.zip", message: 403 - download_cap_exceeded: Cannot download file, download bandwidth or transaction (Class B) cap exceeded. See the Caps & Alerts page to increase your cap.
Exception: 403 - download_cap_exceeded: Cannot download file, download bandwidth or transaction (Class B) cap exceeded. See the Caps & Alerts page to increase your cap.


2025-01-19 05:32:39 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3948.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3948.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3948.NEF"
2025-01-19 05:33:02 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3947.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3947.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3947.NEF"
2025-01-19 05:33:29 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3894.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3894.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3894.NEF"
2025-01-19 05:33:38 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3893.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3893.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3893.NEF"
2025-01-19 05:34:20 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3944.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3944.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3944.NEF"
2025-01-19 05:35:00 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3945.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3945.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3945.NEF"
2025-01-19 05:35:23 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3895.NEF". Error message was: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3895.NEF". File hash is 8+oDxZ4DfILxGFmMOeo4k+sE+pPFUP5jhYUEvzC6L9c=, expected hash is NwtQTnneYuy4RBi839QezPIVzhh6Tfmd9t6yxVJ+lEQ=
Exception: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3895.NEF". File hash is 8+oDxZ4DfILxGFmMOeo4k+sE+pPFUP5jhYUEvzC6L9c=, expected hash is NwtQTnneYuy4RBi839QezPIVzhh6Tfmd9t6yxVJ+lEQ=
2025-01-19 05:35:30 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3943.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3943.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3943.NEF"
2025-01-19 05:36:53 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3892.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3892.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3892.NEF"
2025-01-19 05:37:01 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3891.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3891.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3891.NEF"
2025-01-19 05:37:25 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3889.NEF". Error message was: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3889.NEF". File hash is 0z82FiW+1nT/59SdDlZ2GZ76hexud7wX69yU2CCq80A=, expected hash is LRaPHCv+FZ1hzn4E72zf7pgXO1L+mKLvMzUUz4qp0aA=
Exception: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3889.NEF". File hash is 0z82FiW+1nT/59SdDlZ2GZ76hexud7wX69yU2CCq80A=, expected hash is LRaPHCv+FZ1hzn4E72zf7pgXO1L+mKLvMzUUz4qp0aA=
2025-01-19 05:38:20 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3946.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3946.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3946.NEF"
2025-01-19 05:38:24 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3890.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3890.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3890.NEF"
2025-01-19 05:38:25 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3942.NEF". Error message was: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3942.NEF". File hash is mBemmdlwXIOGZTmSxO4fzvYTyIiNIPoQWNplgApiqbI=, expected hash is LI6YqHPHLYpHAgG+gZQy0ufvq2ur7Dk/Ogw5CPmCOzA=
Exception: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3942.NEF". File hash is mBemmdlwXIOGZTmSxO4fzvYTyIiNIPoQWNplgApiqbI=, expected hash is LI6YqHPHLYpHAgG+gZQy0ufvq2ur7Dk/Ogw5CPmCOzA=
2025-01-19 08:31:18 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3948.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3948.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3948.NEF"
2025-01-19 08:31:43 +00 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: "/BIG1/nextcloud-data/dat/files/photo/DSC_3947.NEF". Error message was: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3947.NEF"
FileNotFoundException: Could not find file "/BIG1/nextcloud-data/dat/files/photo/DSC_3947.NEF"

as you can see there are 2 classes of errors:

  • at around 2025-01-18 23:36:46 I had my daily cost cap hit on Backblaze, when I received the notification I immediately increased it to allow all the data to be downloaded.
  • at 2025-01-19 05:32:39 I got errors on missing files or wrong hash

I’m running Duplicati - 2.0.8.1_beta_2024-05-07 on rasperry pi, I would like to ask if the 2 errors are related (i.e. block failed to download → missing file) or are they unrelated?

Any easy way to have duplicati check the local data against the backup and download the missing files without triggering a full re-download (which takes a good 10 hours)?

Thanks for any explanation.
Arjuna

Welcome to the forum @adeltoso

What sort? “Direct restore from backup files”, or did you make a job and recreate database?

Which ones?

Depending on log, you might not have a complete list of the errors.
This is especially true of Job → Show log, which limits the storage.
Complete log for a job will show the actual totals.

      "MessagesActualLength": #,
      "WarningsActualLength": #,
      "ErrorsActualLength": #,

I’m a little surprised that it kept going in the face of the 403 errors, but it looks like it recognized eventually (when it went to verify the restored files) that it didn’t put all the files back as needed.

Files are not stored directly in the backup, so it’s not exactly a direct download that you’re after.

TECHNICAL DETAILS gives background, for example Understanding Restore starts by saying:

For the restore process, the goal is to recreate the files in a “snapshot”, with the contents as they were at the time the backup was made. To achieve this, the restore process first obtains a list of files, and from that builds a list of blocks that are required to recreate these files.

and the last part of last sentence is key. This isn’t a copy. It gets blocks out of the dblock files as needed for the restore, and it looks first to make sure no extra dblock downloads are requested.

Since you have a partial restore already, doing the same thing again should finish doing restore, however if it was a “Direct restore” there will be some extra initial downloads to create database.

EDIT:

TL;DR is above, but given some uncertainties, does it look like you got a partial restore finished?
It’s sometimes hard to tell. Files are patched together block by block, so partial files are possible.
If you know expected total file size of source, you can check that against what restore did before running against the download cap. This is really only relevant to how much restore is left though.

Thank you! And thank you for your reply.

I clicked “Restore files” there and when I checked back I got the errors in red at the bottom

I found the complete logs

  "MessagesActualLength": 11860,
  "WarningsActualLength": 22,
  "ErrorsActualLength": 32,

and indeed I’m missing entries as it seems the logs only retain 20 rows. So I’ll be missing 12 errors and 23 warnings :frowning:

I did read the docs you shared, I couldn’t find what I would really need in my case: a cli command to compare local data to the backup version (even just at the duplciati DB level) and report differences. Maybe I missed the relevant doc page or maybe it doesn’t exist, do you happen to know?

I forgot exact number, so I had to find it.

So you’re slightly in the dark, but I don’t know how big a problem it will be. If you really want:

log-file=<path> and log-file-log-level=warning in your Advanced options can get all.

I don’t think there is purely that, but what you describe is roughly what restore does at its end:

Since you have a job set up and seemingly OK, why don’t you just Restore exactly as before?
Whatever is already as it should be will be left alone. Whatever is not done will get completed.

If you’re just hoping that the original restore completed despite errors, you can pick one of the complaint files, say /BIG1/nextcloud-data/dat/files/photo/DSC_3948.NEF and look in the Restore tree (or use the Search for files). Is it in backup? Is it in the restore area already?

Thanks for all the help. I re-ran it and it completed with no Errors (albeit it had 1k warnings on symlinks, but I might have caused that). I think I was able to restore all the data. Long live Duplicati!

2 Likes