@ts678 deserves all the credit for this. I was just the messenger.
@ts678 gets all the credit for this one, too!
Sorry @ts678, I looked at the merge commit names.
Both were collaborations. @warwickmm confirmed the failure, did the pull request, rushing for a canary, and helped me enough that I did the upload throttling pull request myself. @drwtsn32 ran with findings I gave, which is what I always hope for. Thanks to both of them. I liked the flow, even if credits get hazy. 
In testing news, I ran about 127 backups, and Duplicati survived a number of odd B2 errors (which I might ask Backblaze about), but raised/emailed DeleteRemoteFileFailed Warnings when it tried a blind (no List) Delete of dlist after Put failed. Failed dblocks went OK. The only runs that turned yellow in the new job log were the ones with Delete failures. A side issue was Warnings section only found Full log warnings once.
Other Put errors found by an extended regular expression search for RetryPut|DeleteRemoteFileFailed in the --profile-all-sql-queries log I had set. That seemed the nicest way to get glogg to pull summary below:
2019-06-30 20:00:54 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T000000Z.dlist.zip.aes attempt 1 of 5 failed with message: Unable to connect to the remote server
2019-06-30 20:01:07 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T000000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 20:01:07 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T000000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 20:01:07 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T000000Z.dlist.zip.aes with FileNotFound, listing contents
2019-06-30 21:40:56 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b52590d8f4b29437e846e9d2f0daae04c.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-06-30 22:21:04 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T022000Z.dlist.zip.aes attempt 1 of 5 failed with message: 500 - internal_error: incident id 9f5a7c81f0f5-6a4950e40060
2019-06-30 22:21:16 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T022001Z.dlist.zip.aes attempt 2 of 5 failed with message: 503 - service_unavailable: c001_v0001117_t0006 is too busy
2019-06-30 22:21:29 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022000Z.dlist.zip.aes with FileNotFound, listing contents
2019-06-30 22:21:30 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022001Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 22:21:29 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022000Z.dlist.zip.aes with FileNotFound, listing contents,
C: 2019-06-30 22:21:30 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022001Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 22:21:29 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 22:21:30 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022001Z.dlist.zip.aes with FileNotFound, listing contents
2019-07-01 02:10:37 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T061000Z.dlist.zip.aes attempt 1 of 5 failed with message: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
2019-07-01 02:10:52 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-ba5b657f0463d47c48ce4b424ac7bc689.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 02:11:35 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T061000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 02:11:35 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T061000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 02:11:35 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T061000Z.dlist.zip.aes with FileNotFound, listing contents
2019-07-01 02:20:48 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-bd7a3582195664bd9b43a2a45402a3095.dblock.zip.aes attempt 1 of 5 failed with message: 500 - internal_error: incident id 9f5a7c81f0f5-8ab04440d2b0
2019-07-01 04:41:13 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-ba6b2ab432ab941e78a6a3f60f02dc556.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 04:41:47 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b7a8ae315913440ca9774537628b0d3ba.dblock.zip.aes attempt 2 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 09:40:36 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T134000Z.dlist.zip.aes attempt 1 of 5 failed with message: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
2019-07-01 09:40:53 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b0206909db19241208aa9281e05b1bf26.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 09:41:51 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T134000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 09:41:51 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T134000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 09:41:51 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T134000Z.dlist.zip.aes with FileNotFound, listing contents
2019-07-01 14:10:53 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-bf8d3b5103efb4562839ee029e80d1d2e.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 15:30:44 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-bab8687c06920471fa274eab2751dc42d.dblock.zip.aes attempt 1 of 5 failed with message: 503 - service_unavailable: c001_v0001116_t0042 is too busy
2019-07-01 18:30:38 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T223000Z.dlist.zip.aes attempt 1 of 5 failed with message: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
2019-07-01 18:31:00 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b9f2120edcc6e4e85b31ca299c3215925.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 18:31:47 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T223000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 18:31:47 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T223000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 18:31:47 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T223000Z.dlist.zip.aes with FileNotFound, listing contents
Nothing seen so far is a show-stopper (though I dislike the noise when B2 fails), and endurance is hugely better than it would have been without the fix for data corruption during compacting. Will continue tests…
Hi,
I’m using You are currently running Duplicati - 2.0.4.20_canary_2019-06-25 with upload throttling, how is affecting this the backups causing the data corruption? are my backup corrupted since this was fixed until 2.0.4.22?
Also keen to know this!
TL;DR Unless you set unusual settings, you’d likely have noticed any issues if you’ve been running awhile.
Upload throttle corrupts backup, especially OneDrive. Analyzed, with code proposed. #3787 describes the issue, and there wasn’t an exhaustive study of which backends were more affected. It’s unpredictable, but also an extremely old bug. It would show up on the default automatic verification eventually, but due to that small sample (one set), you can also run the test command in Commandline, using a higher sample size.
–backup-test-samples can also raise the sample size of the default automatic verification if you prefer that.
A third option especially well suited to destinations on a system with PowerShell or Python is to set option –upload-verification-file=true then go to your installation’s utility-scripts folder to run DuplicatiVerify.{ps1,py}.