Release: 2.0.4.22 (canary) 2019-06-30

Both were collaborations. @warwickmm confirmed the failure, did the pull request, rushing for a canary, and helped me enough that I did the upload throttling pull request myself. @drwtsn32 ran with findings I gave, which is what I always hope for. Thanks to both of them. I liked the flow, even if credits get hazy. :wink:

In testing news, I ran about 127 backups, and Duplicati survived a number of odd B2 errors (which I might ask Backblaze about), but raised/emailed DeleteRemoteFileFailed Warnings when it tried a blind (no List) Delete of dlist after Put failed. Failed dblocks went OK. The only runs that turned yellow in the new job log were the ones with Delete failures. A side issue was Warnings section only found Full log warnings once.

Other Put errors found by an extended regular expression search for RetryPut|DeleteRemoteFileFailed in the --profile-all-sql-queries log I had set. That seemed the nicest way to get glogg to pull summary below:

2019-06-30 20:00:54 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T000000Z.dlist.zip.aes attempt 1 of 5 failed with message: Unable to connect to the remote server
2019-06-30 20:01:07 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T000000Z.dlist.zip.aes with FileNotFound, listing contents
C:     2019-06-30 20:01:07 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T000000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 20:01:07 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T000000Z.dlist.zip.aes with FileNotFound, listing contents


2019-06-30 21:40:56 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b52590d8f4b29437e846e9d2f0daae04c.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.


2019-06-30 22:21:04 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T022000Z.dlist.zip.aes attempt 1 of 5 failed with message: 500 - internal_error: incident id 9f5a7c81f0f5-6a4950e40060
2019-06-30 22:21:16 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T022001Z.dlist.zip.aes attempt 2 of 5 failed with message: 503 - service_unavailable: c001_v0001117_t0006 is too busy
2019-06-30 22:21:29 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022000Z.dlist.zip.aes with FileNotFound, listing contents
2019-06-30 22:21:30 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022001Z.dlist.zip.aes with FileNotFound, listing contents
C:     2019-06-30 22:21:29 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022000Z.dlist.zip.aes with FileNotFound, listing contents,
C:     2019-06-30 22:21:30 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022001Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 22:21:29 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-06-30 22:21:30 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T022001Z.dlist.zip.aes with FileNotFound, listing contents


2019-07-01 02:10:37 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T061000Z.dlist.zip.aes attempt 1 of 5 failed with message: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.


2019-07-01 02:10:52 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-ba5b657f0463d47c48ce4b424ac7bc689.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 02:11:35 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T061000Z.dlist.zip.aes with FileNotFound, listing contents
C:     2019-07-01 02:11:35 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T061000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 02:11:35 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T061000Z.dlist.zip.aes with FileNotFound, listing contents


2019-07-01 02:20:48 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-bd7a3582195664bd9b43a2a45402a3095.dblock.zip.aes attempt 1 of 5 failed with message: 500 - internal_error: incident id 9f5a7c81f0f5-8ab04440d2b0


2019-07-01 04:41:13 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-ba6b2ab432ab941e78a6a3f60f02dc556.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 04:41:47 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b7a8ae315913440ca9774537628b0d3ba.dblock.zip.aes attempt 2 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.


2019-07-01 09:40:36 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T134000Z.dlist.zip.aes attempt 1 of 5 failed with message: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
2019-07-01 09:40:53 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b0206909db19241208aa9281e05b1bf26.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 09:41:51 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T134000Z.dlist.zip.aes with FileNotFound, listing contents
C:     2019-07-01 09:41:51 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T134000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 09:41:51 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T134000Z.dlist.zip.aes with FileNotFound, listing contents


2019-07-01 14:10:53 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-bf8d3b5103efb4562839ee029e80d1d2e.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.


2019-07-01 15:30:44 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-bab8687c06920471fa274eab2751dc42d.dblock.zip.aes attempt 1 of 5 failed with message: 503 - service_unavailable: c001_v0001116_t0042 is too busy


2019-07-01 18:30:38 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20190701T223000Z.dlist.zip.aes attempt 1 of 5 failed with message: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
2019-07-01 18:31:00 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b9f2120edcc6e4e85b31ca299c3215925.dblock.zip.aes attempt 1 of 5 failed with message: Unable to read data from the transport connection: The connection was closed.
2019-07-01 18:31:47 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T223000Z.dlist.zip.aes with FileNotFound, listing contents
C:     2019-07-01 18:31:47 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T223000Z.dlist.zip.aes with FileNotFound, listing contents
C: 2019-07-01 18:31:47 -04 - [Warning-Duplicati.Library.Main.BackendManager-DeleteRemoteFileFailed]: Delete operation failed for duplicati-20190701T223000Z.dlist.zip.aes with FileNotFound, listing contents

Nothing seen so far is a show-stopper (though I dislike the noise when B2 fails), and endurance is hugely better than it would have been without the fix for data corruption during compacting. Will continue tests…