Backup to B2 but no files sent

Remsky · August 20, 2020, 8:15pm

Hello,

I’ve just “succeeded” in configuring BackBlaze B2 as the back-end, successfully tested the connection and tried a small folder which claims to have run, but logging in to BackBlaze show’s nothing there…

Duplicati however offers a restore point too, with a full file index

HOWEVER
when I’ve tried to re-run the test, or restore a file I do get an error message:
“Found 3 files that are missing from the remote storage, please run repair”

the log shows all green with no warnings or errors

Hoping someone can point out my rookie mistake

running Manjaro

Many thanks in advance

{
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 68,
“OpenedFiles”: 68,
“AddedFiles”: 68,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 43641032,
“SizeOfExaminedFiles”: 43641032,
“SizeOfOpenedFiles”: 43641032,
“NotProcessedFiles”: 0,
“AddedFolders”: 8,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: null,
“VacuumResults”: null,
“DeleteResults”: null,
“RepairResults”: null,
“TestResults”: {
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20200820T195259Z.dlist.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-i5aa2ea9f19234f7e8fddbc5d47c5da93.dindex.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-b072caefd79d1434e9875ac766ebcb1b9.dblock.zip.aes”,
“Value”:
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “2020-08-20T19:54:07.694994Z”,
“BeginTime”: “2020-08-20T19:53:52.241444Z”,
“Duration”: “00:00:15.4535500”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 42596071,
“BytesDownloaded”: 42596071,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 3,
“KnownFileSize”: 42596071,
“LastBackupDate”: “2020-08-20T20:52:59+01:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-08-20T19:52:58.877182Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “2020-08-20T19:54:07.713052Z”,
“BeginTime”: “2020-08-20T19:52:58.875859Z”,
“Duration”: “00:01:08.8371930”,
“MessagesActualLength”: 17,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: [
“2020-08-20 20:52:59 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started”,
“2020-08-20 20:52:59 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2020-08-20 20:53:00 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: ()”,
“2020-08-20 20:53:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b072caefd79d1434e9875ac766ebcb1b9.dblock.zip.aes (40.57 MB)”,
“2020-08-20 20:53:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20200820T195259Z.dlist.zip.aes (9.95 KB)”,
“2020-08-20 20:53:12 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20200820T195259Z.dlist.zip.aes (9.95 KB)”,
“2020-08-20 20:53:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b072caefd79d1434e9875ac766ebcb1b9.dblock.zip.aes (40.57 MB)”,
“2020-08-20 20:53:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i5aa2ea9f19234f7e8fddbc5d47c5da93.dindex.zip.aes (44.42 KB)”,
“2020-08-20 20:53:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i5aa2ea9f19234f7e8fddbc5d47c5da93.dindex.zip.aes (44.42 KB)”,
“2020-08-20 20:53:50 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2020-08-20 20:53:52 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (3 bytes)”,
“2020-08-20 20:53:52 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200820T195259Z.dlist.zip.aes (9.95 KB)”,
“2020-08-20 20:53:53 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200820T195259Z.dlist.zip.aes (9.95 KB)”,
“2020-08-20 20:53:53 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-i5aa2ea9f19234f7e8fddbc5d47c5da93.dindex.zip.aes (44.42 KB)”,
“2020-08-20 20:53:54 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-i5aa2ea9f19234f7e8fddbc5d47c5da93.dindex.zip.aes (44.42 KB)”,
“2020-08-20 20:53:54 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b072caefd79d1434e9875ac766ebcb1b9.dblock.zip.aes (40.57 MB)”,
“2020-08-20 20:54:07 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b072caefd79d1434e9875ac766ebcb1b9.dblock.zip.aes (40.57 MB)”
],
“Warnings”: ,
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 42596071,
“BytesDownloaded”: 42596071,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 3,
“KnownFileSize”: 42596071,
“LastBackupDate”: “2020-08-20T20:52:59+01:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-08-20T19:52:58.877182Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

drwtsn32 · August 20, 2020, 10:19pm

What does your “Folder path” look like on the Destination section of your backup job? Can you paste the exact string here? If it has sensitive info feel free to change letters but don’t adjust any symbols or slashes.

Remsky · August 21, 2020, 3:10pm

Hi
Thanks for looking into this with me

Target URL looks like this:
b2://Test-Art/?auth-username=B2ApplicationID&auth-password=B2ApplicationKey

The bucket “Test-Art” exists in BackBlaze and it has 3 files prefixed “duplicati-”
.dlist.zip.aes [10kb]
.dblock.zip.aes [42MB]
.dindex.zip.aes [45MB]

drwtsn32 · August 21, 2020, 3:13pm

On page 2 of your backup configuration, is the “Folder path” blank?

Remsky · August 22, 2020, 1:13pm

yes it is blank, and the 3 .aes files are saved to the bucket root.

When I try with a folder it instantly fails: “Found 3 files that are missing from the remote storage, please run repair”

drwtsn32 · August 22, 2020, 1:28pm

I’m not sure if having a blank is your problem or not. I’ll try to test that scenario.

But yes, if you add a folder now it will not see the three files in there. If you are only just starting and are willing to redo your initial backup, then try deleting the three duplicati files in the root of your bucket, and then deleting the local job database. (Click on the job in the web UI, click database, then click delete.)

Then when you re-run your backup it should “start over” and upload the files to the subfolder in the bucket. See if that works better! If it does then there’s a bug when you try using the root of a bucket.

Remsky · August 22, 2020, 5:11pm

OK - Dr Watson, you are a genius!

I tried that (deleted files & database & re-run)
I now have 4 files in the bucket folder (3 x .aes + .bzEmpty)
Tested a single file restore which appears to have worked too !!!

Thank you

Last successful backup: Today at 17:38 (took 00:01:10)
Log:
{
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 68,
“OpenedFiles”: 68,
“AddedFiles”: 68,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 43641032,
“SizeOfExaminedFiles”: 43641032,
“SizeOfOpenedFiles”: 43641032,
“NotProcessedFiles”: 0,
“AddedFolders”: 8,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: null,
“VacuumResults”: null,
“DeleteResults”: null,
“RepairResults”: null,
“TestResults”: {
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20200822T163719Z.dlist.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-icf0e545bb34b42769fd7dca11d705364.dindex.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-bf09d913e58ba444fbdf35436330fd433.dblock.zip.aes”,
“Value”:
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “2020-08-22T16:38:29.063267Z”,
“BeginTime”: “2020-08-22T16:38:11.430886Z”,
“Duration”: “00:00:17.6323810”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 42607655,
“BytesDownloaded”: 42607655,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 1,
“KnownFileCount”: 3,
“KnownFileSize”: 42607655,
“LastBackupDate”: “2020-08-22T17:37:19+01:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-08-22T16:37:19.129903Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “2020-08-22T16:38:29.075505Z”,
“BeginTime”: “2020-08-22T16:37:19.129898Z”,
“Duration”: “00:01:09.9456070”,
“MessagesActualLength”: 17,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: [
“2020-08-22 17:37:19 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started”,
“2020-08-22 17:37:19 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2020-08-22 17:37:20 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (1 bytes)”,
“2020-08-22 17:37:27 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bf09d913e58ba444fbdf35436330fd433.dblock.zip.aes (40.58 MB)”,
“2020-08-22 17:37:27 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20200822T163719Z.dlist.zip.aes (9.95 KB)”,
“2020-08-22 17:37:29 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20200822T163719Z.dlist.zip.aes (9.95 KB)”,
“2020-08-22 17:38:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bf09d913e58ba444fbdf35436330fd433.dblock.zip.aes (40.58 MB)”,
“2020-08-22 17:38:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-icf0e545bb34b42769fd7dca11d705364.dindex.zip.aes (44.43 KB)”,
“2020-08-22 17:38:10 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-icf0e545bb34b42769fd7dca11d705364.dindex.zip.aes (44.43 KB)”,
“2020-08-22 17:38:10 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2020-08-22 17:38:11 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (4 bytes)”,
“2020-08-22 17:38:11 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200822T163719Z.dlist.zip.aes (9.95 KB)”,
“2020-08-22 17:38:12 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200822T163719Z.dlist.zip.aes (9.95 KB)”,
“2020-08-22 17:38:12 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-icf0e545bb34b42769fd7dca11d705364.dindex.zip.aes (44.43 KB)”,
“2020-08-22 17:38:13 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-icf0e545bb34b42769fd7dca11d705364.dindex.zip.aes (44.43 KB)”,
“2020-08-22 17:38:13 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bf09d913e58ba444fbdf35436330fd433.dblock.zip.aes (40.58 MB)”,
“2020-08-22 17:38:29 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bf09d913e58ba444fbdf35436330fd433.dblock.zip.aes (40.58 MB)”
],
“Warnings”: ,
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 42607655,
“BytesDownloaded”: 42607655,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 1,
“KnownFileCount”: 3,
“KnownFileSize”: 42607655,
“LastBackupDate”: “2020-08-22T17:37:19+01:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.110 (2.0.5.110_canary_2020-08-10)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-08-22T16:37:19.129903Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

drwtsn32 · August 22, 2020, 5:46pm

Great! Glad that was the solution. Seems like a bug in Duplicati though that it can’t handle the root of the bucket.

In any case I think using folders is a better idea. This way you can use the same bucket for multiple backup jobs on one or more computers. (Each backup job on each machine should target a unique path so the back end files don’t get mixed together.)

ts678 · August 22, 2020, 7:50pm

Possibly related to the occasional problems with extra slash, e.g. at end of folder.

733 Files missing at the remote locations but all are there

Backup files missing on Wasabi was a case where Wasabi’s viewer didn’t see…
Cyberduck also didn’t, and the reason may be from slash handling in object keys.

Behavior might vary depending on destination. Fix mustn’t harm existing backups.