Missing files on remote

ts678 · October 7, 2025, 4:07pm

Below is likely reference. Discussion followed by some, but I don’t think dev commented.

2025-08-15 03:41:46 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-ResponseHeadersStop]: [System.Net.Http] Event: ResponseHeadersStop
  statusCode          : 503

github.com/duplicati/duplicati

Backblaze B2 uploads fail, often with timeout. Big read-write-timeout may help.

opened 12:16AM - 29 Jul 25 UTC

closed 01:21PM - 04 Aug 25 UTC

ts678

- [x] I have searched open and closed issues for duplicates. - [x] I have search…ed the [forum](https://forum.duplicati.com) for related topics. ---------------------------------------- ## Environment info - **Duplicati version**: 2.1.1.0 Experimental and roughly Canary 2.1.0.102-2.1.0.125 so far. - **Operating system**: Windows or Linux - **Backend**: Backblaze B2 ## Description Backblaze B2 has had read-write-timeout upload failures since 2.1.0.102 (hard to pinpoint). Default 30 second timeout is not solid enough, so I'm testing with 15 second timeout now. See [Release: 2.1.0.125 (Canary) 2025-07-15](https://forum.duplicati.com/t/release-2-1-0-125-canary-2025-07-15/20889), and another user where 5 minute timeout fails. This seems to affect the B2 backend, not Backblaze S3. Early contrary reports seem in error. 2.1.0.125 with BackendTool: ``` Command failed: The operation has timed out. System.TimeoutException: The operation has timed out. at Duplicati.StreamUtil.TimeoutObservingStream.ReadImplAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken) at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, String filename, CancellationToken cancelToken) at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, String filename, CancellationToken cancelToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTool.Program.Main(String[] _args) ``` 2.1.0.125 with BackendTester: ``` 7/28/2025 6:54:45 PM [18:54:45 866] Starting run no 0 [18:54:48 449] Read Write Timeout set to 60000 ms [18:54:48 455] Generating file 0 (8.50 MiB) [18:54:48 593] Generating file 1 (14.18 MiB) [18:54:48 796] Generating file 2 (6.32 MiB) [18:54:48 897] Generating file 3 (29.08 MiB) [18:54:49 299] Generating file 4 (10.70 MiB) [18:54:49 446] Generating file 5 (21.17 MiB) [18:54:49 726] Generating file 6 (24.93 MiB) [18:54:50 038] Generating file 7 (42.42 MiB) [18:54:50 563] Generating file 8 (34.36 MiB) [18:54:50 987] Generating file 9 (14.75 MiB) [18:54:51 165] Uploading wrong files ... [18:54:51 165] Generating file 10 (1.71 KiB) [18:54:51 199] Uploading file 0, 1.71 KiB ... done! in 406 ms (~4.22 KiB/s) [18:54:51 606] Uploading file 0, 1.71 KiB ... done! in 496 ms (~3.45 KiB/s) [18:54:52 102] Uploading file 9, 1.71 KiB ... done! in 194 ms (~8.81 KiB/s) [18:54:52 298] Uploading files ... [18:54:52 298] Uploading file 0, 8.50 MiB ... done! in 7449 ms (~1.14 MiB/s) [18:54:59 749] Uploading file 1, 14.18 MiB ... done! in 6663 ms (~2.13 MiB/s) [18:55:06 412] Uploading file 2, 6.32 MiB ... done! in 6612 ms (~978.08 KiB/s) [18:55:13 025] Uploading file 3, 29.08 MiB ... [18:55:28 208] Failed to upload file 3, error message: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming, Int64 throttle, Int32 readWriteTimeout), remote name: yMsjSpVqmA54vQeb5qNhSsO5cogN8x5YKBv3F3z5jNUkb7GjUBa3Qf49NLVoGLY57kWEI6n7q after 15182 ms [18:55:28 248] Inner exception: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) [18:55:28 256] Inner exception: System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [18:55:28 258] Inner exception: System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) [18:55:28 259] Inner exception: System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. [18:55:28 260] Uploading file 4, 10.70 MiB ... done! in 7031 ms (~1.52 MiB/s) [18:55:35 291] Uploading file 5, 21.17 MiB ... done! in 15811 ms (~1.34 MiB/s) [18:55:51 105] Uploading file 6, 24.93 MiB ... [18:56:06 235] Failed to upload file 6, error message: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming, Int64 throttle, Int32 readWriteTimeout), remote name: ORPi9JbR54Z41U2Xd6JEwOr3VziiQQFER after 15130 ms [18:56:06 238] Inner exception: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) [18:56:06 239] Inner exception: System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [18:56:06 240] Inner exception: System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) [18:56:06 241] Inner exception: System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. [18:56:06 241] Uploading file 7, 42.42 MiB ... [18:56:21 889] Failed to upload file 7, error message: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming, Int64 throttle, Int32 readWriteTimeout), remote name: 88opqwalpcBhewMnPVv1KrvZP46OEzB3Dcyvg7oi3DaO92ZYuVI after 15648 ms [18:56:21 893] Inner exception: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) [18:56:21 895] Inner exception: System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [18:56:21 897] Inner exception: System.IO.IOException: Unable to write data to the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) [18:56:21 899] Inner exception: System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request. [18:56:21 900] Uploading file 8, 34.36 MiB ... [18:56:37 515] Failed to upload file 8, error message: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming, Int64 throttle, Int32 readWriteTimeout), remote name: jWWqsltgZkA after 15615 ms [18:56:37 516] Inner exception: System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) [18:56:37 519] Inner exception: System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) [18:56:37 520] Inner exception: System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1[TIOAdapter](ValueTask writeTask, Byte[] bufferToReturn) at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task copyTask, Stream source) at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) [18:56:37 520] Inner exception: System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. [18:56:37 520] Uploading file 9, 14.75 MiB ... done! in 8901 ms (~1.66 MiB/s) [18:56:46 423] Verifying file list ... [18:56:46 520] *** File 3 with name yMsjSpVqmA54vQeb5qNhSsO5cogN8x5YKBv3F3z5jNUkb7GjUBa3Qf49NLVoGLY57kWEI6n7q was uploaded but not found afterwards [18:56:46 520] *** File 6 with name ORPi9JbR54Z41U2Xd6JEwOr3VziiQQFER was uploaded but not found afterwards [18:56:46 520] *** File 7 with name 88opqwalpcBhewMnPVv1KrvZP46OEzB3Dcyvg7oi3DaO92ZYuVI was uploaded but not found afterwards [18:56:46 520] *** File 8 with name jWWqsltgZkA was uploaded but not found afterwards [18:56:46 520] Downloading files [18:56:46 522] Downloading file 0 ... done in 2960 ms (~2.87 MiB/s) [18:56:49 483] Checking hash ... done [18:56:49 560] Downloading file 1 ... done in 4088 ms (~3.47 MiB/s) [18:56:53 649] Checking hash ... done [18:56:53 777] Downloading file 2 ... done in 1662 ms (~3.80 MiB/s) [18:56:55 440] Checking hash ... done [18:56:55 515] Downloading file 3 ... [18:56:55 948] failed *** Error: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.GetAsync(String remotename, Stream stream, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) after 432 ms [18:56:55 949] Checking hash ... [18:56:55 949] failed *** Downloaded file was corrupt [18:56:55 951] Downloading file 4 ... done in 3459 ms (~3.09 MiB/s) [18:56:59 412] Checking hash ... done [18:56:59 518] Downloading file 5 ... done in 5711 ms (~3.71 MiB/s) [18:57:05 249] Checking hash ... done [18:57:05 439] Downloading file 6 ... [18:57:05 951] failed *** Error: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.GetAsync(String remotename, Stream stream, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) after 511 ms [18:57:05 954] Checking hash ... [18:57:05 954] failed *** Downloaded file was corrupt [18:57:05 955] Downloading file 7 ... [18:57:06 134] failed *** Error: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.GetAsync(String remotename, Stream stream, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) after 179 ms [18:57:06 134] Checking hash ... [18:57:06 135] failed *** Downloaded file was corrupt [18:57:06 136] Downloading file 8 ... [18:57:06 314] failed *** Error: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.GetAsync(String remotename, Stream stream, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) after 177 ms [18:57:06 314] Checking hash ... [18:57:06 315] failed *** Downloaded file was corrupt [18:57:06 315] Downloading file 9 ... done in 4496 ms (~3.28 MiB/s) [18:57:10 813] Checking hash ... done [18:57:10 945] Deleting files... [18:57:10 946] Deleting file 0 [18:57:11 088] Deleting file 1 [18:57:11 181] Deleting file 2 [18:57:11 272] Deleting file 3 [18:57:11 369] *** Failed to delete file yMsjSpVqmA54vQeb5qNhSsO5cogN8x5YKBv3F3z5jNUkb7GjUBa3Qf49NLVoGLY57kWEI6n7q, message: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.DeleteAsync(String remotename, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) [18:57:11 370] Deleting file 4 [18:57:11 558] Deleting file 5 [18:57:11 651] Deleting file 6 [18:57:11 763] *** Failed to delete file ORPi9JbR54Z41U2Xd6JEwOr3VziiQQFER, message: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.DeleteAsync(String remotename, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) [18:57:11 764] Deleting file 7 [18:57:11 860] *** Failed to delete file 88opqwalpcBhewMnPVv1KrvZP46OEzB3Dcyvg7oi3DaO92ZYuVI, message: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.DeleteAsync(String remotename, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) [18:57:11 861] Deleting file 8 [18:57:11 955] *** Failed to delete file jWWqsltgZkA, message: Duplicati.Library.Interface.FileMissingException: The requested file does not exist at Duplicati.Library.Backend.Backblaze.B2.DeleteAsync(String remotename, CancellationToken cancellationToken) at Duplicati.Library.Utility.Utility.Await(Task task) at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first) [18:57:11 955] Deleting file 9 [18:57:12 514] Checking retrieval of non-existent file... [18:57:12 695] Caught expected FileMissingException [18:57:12 727] Checking DNS names used by this backend... api.backblazeb2.com api001.backblazeb2.com f001.backblazeb2.com ``` ## Steps to reproduce 1. Using 15 second read-write-timeout, BackendTool put a 40 MB file, or run a cycle of BackendTester. - **Actual result**: It might fail. BackendTool usually says TimeoutException. BackendTester makes a different messy stack. - **Expected result**: Upload works. ## Screenshots ## Debug log

was said (by GitHub version note) to be fixed in 2.1.1.100 by

Removing chunksize from B2 backend #6441

and yet it seems like it was seen in 2.1.1.100 in quote above, along with the old note:

I, for one, would be interested in looking deeper. There are also ideas at:

2.1.1.104 > Files are missing from the remote storage

One unanswered question is whether missing files got a statusCode 200.

Since the previous problem is read-write-timeout, does increasing it help?

If there is an unusual remote volume size or advanced options, mention it.

Running BackendTester would be good. In early worse days, that got odd.