Wasabi ReadDoneAsync2 Error

papaxris · July 26, 2023, 1:38pm

Hello Duplicati Community,

A recent error has me concerned about my remote backups to Wasabi. Local network backups using Duplicati continue to work.

I have been using Duplicati for years and have been running recently version 2.0.7.1_beta_2023-05-25 without issues.

Yesterday I started to receive the error listed below for all my remote backups to Wasabi. No configuration changes were made to Wasabi or Duplicati, which makes me think perhaps something with Wasabi endpoint is affecting the backups.

Any assistance or pointers in the right direction would be greatly appreciated. Thanks in advance for your help, please let me know how I can assist if more detail is needed.

Failed: A WebException with status ReceiveFailure was thrown.
Details: Amazon.Runtime.AmazonServiceException: A WebException with status ReceiveFailure was thrown. ---> System.Net.WebException: Error getting response stream (ReadDoneAsync2): ReceiveFailure
  at System.Net.WebResponseStream.InitReadAsync (System.Threading.CancellationToken cancellationToken) [0x000f3] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebOperation.Run () [0x001d9] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003b] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <bd464cdb92394af6a8e27f353fa19333>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <1d5e23fb32014e869fdd576305c21e36>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.BackendManager.List () [0x00049] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x0000d] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00000] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0005f] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01048] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011c] in <9cab5b8f1b4f49ec980acae8e278968b>:0 

Log data:
2023-07-26 11:02:45 +00 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Amazon.Runtime.AmazonServiceException: A WebException with status ReceiveFailure was thrown. ---> System.Net.WebException: Error getting response stream (ReadDoneAsync2): ReceiveFailure
  at System.Net.WebResponseStream.InitReadAsync (System.Threading.CancellationToken cancellationToken) [0x000f3] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebOperation.Run () [0x001d9] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003b] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <bd464cdb92394af6a8e27f353fa19333>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <1d5e23fb32014e869fdd576305c21e36>:0 
  at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <1d5e23fb32014e869fdd576305c21e36>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.BackendManager.List () [0x00049] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x0000d] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00000] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0005f] in <9cab5b8f1b4f49ec980acae8e278968b>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00452] in <9cab5b8f1b4f49ec980acae8e278968b>:0

papaxris · July 27, 2023, 2:39pm

Out of curiosity are any other Duplicati users having issues with remote backups to Wasabi endpoints?

I’d like to verify if this is an isolated issue or across the board. Thanks!

Lerrissirrel · July 27, 2023, 3:25pm

Hi papaxris,

I use Wasabi on the east coast of US. It’s still working ok for me.

What OS are you running Duplicati on? I’d searched around on your issue yesterday and didn’t find anything very useful (so didn’t post). But I did find some various mentions that it could have something to do with Mono.

L

papaxris · July 27, 2023, 4:53pm

@Lerrissirrel
Thanks for the heads-up, much appreciated. I’m using Ubuntu server OS for my setup.

I’ll keep researching as well and investigate a bit more into Mono.