Can't upload to BackBlaze: "The channel "BackendRequests" is retired "

u2615 · January 4, 2020, 10:07pm

Hi ! I’m trying to use Duplicati on an Ubuntu VPS Server, to no avail…

Ubuntu Server 18.04.3 LTS
Mono Version 6.6.0.161
Duplicati version 2.0.5.0-1
BackBlaze B2 Cloud Storage with Application Key credentials set up

When uploading, Duplicati keeps on failing with error:

  Listing remote folder ...
  Uploading file (49.25 MB) ...
  Uploading file (49.25 MB) ...
  Uploading file (49.25 MB) ...
  Uploading file (49.12 MB) ...
  Uploading file (1.84 MB) ...
  Uploading file (391.62 KB) ...
  Uploading file (49.25 MB) ...
  Uploading file (49.12 MB) ...
  Uploading file (1.84 MB) ...
  Uploading file (391.62 KB) ...
  Uploading file (49.25 MB) ...
  Uploading file (49.12 MB) ...
  Uploading file (391.62 KB) ...
  Uploading file (1.84 MB) ...
  Uploading file (49.25 MB) ...

System.AggregateException: One or more errors occurred. (The channel "BackendRequests" is retired (The channel "BackendRequests" is retired) (One or more errors occurred. (Cannot access a disposed object.
Object name: 'Stream has been closed'.))) ---> System.AggregateException: The channel "BackendRequests" is retired (The channel "BackendRequests" is retired) (One or more errors occurred. (Cannot access a disposed object.
Object name: 'Stream has been closed'.)) ---> CoCoL.RetiredException: The channel "BackendRequests" is retired
  at CoCoL.Channel`1[T].WriteAsync (T value, CoCoL.ITwoPhaseOffer offer) [0x00641] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x0009d] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00ca8] in <e7bec42ecb504e55951d31db4472cf8b>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <e7bec42ecb504e55951d31db4472cf8b>:0 
   --- End of inner exception stack trace ---
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x0005d] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00074] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.CommandLine.Commands.Backup (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x00119] in <afefaa9be3984a48a07e6d450a8404e7>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00342] in <afefaa9be3984a48a07e6d450a8404e7>:0 
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <afefaa9be3984a48a07e6d450a8404e7>:0 
---> (Inner Exception #0) System.AggregateException: The channel "BackendRequests" is retired (The channel "BackendRequests" is retired) (One or more errors occurred. (Cannot access a disposed object.
Object name: 'Stream has been closed'.)) ---> CoCoL.RetiredException: The channel "BackendRequests" is retired
  at CoCoL.Channel`1[T].WriteAsync (T value, CoCoL.ITwoPhaseOffer offer) [0x00641] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x0009d] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00ca8] in <e7bec42ecb504e55951d31db4472cf8b>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <e7bec42ecb504e55951d31db4472cf8b>:0 
---> (Inner Exception #0) CoCoL.RetiredException: The channel "BackendRequests" is retired
  at CoCoL.Channel`1[T].WriteAsync (T value, CoCoL.ITwoPhaseOffer offer) [0x00641] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x0009d] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00ca8] in <e7bec42ecb504e55951d31db4472cf8b>:0 <---

---> (Inner Exception #1) System.AggregateException: One or more errors occurred. (Cannot access a disposed object.
Object name: 'Stream has been closed'.) ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Stream has been closed'.
  at System.IO.FileStream.BeginRead (System.Byte[] array, System.Int32 offset, System.Int32 numBytes, System.AsyncCallback userCallback, System.Object stateObject) [0x0000d] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.Stream+<>c.<BeginEndReadAsync>b__45_0 (System.IO.Stream stream, System.IO.Stream+ReadWriteParameters args, System.AsyncCallback callback, System.Object state) [0x00000] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncTrim[TInstance,TArgs] (TInstance thisRef, TArgs args, System.Func`5[T1,T2,T3,T4,TResult] beginMethod, System.Func`3[T1,T2,TResult] endMethod) [0x00008] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.Stream.BeginEndReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x0001b] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.Stream.ReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count, System.Threading.CancellationToken cancellationToken) [0x00009] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.FileStream.ReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count, System.Threading.CancellationToken cancellationToken) [0x00000] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at Duplicati.Library.Utility.Utility.CopyStreamAsync (System.IO.Stream source, System.IO.Stream target, System.Boolean tryRewindSource, System.Threading.CancellationToken cancelToken, System.Byte[] buf) [0x0005e] in <18a4f5320dc54e9fa1cd255415a084f9>:0 
  at Duplicati.Library.Utility.Utility.CopyStreamAsync (System.IO.Stream source, System.IO.Stream target, System.Threading.CancellationToken cancelToken) [0x00077] in <18a4f5320dc54e9fa1cd255415a084f9>:0 
  at Duplicati.Library.Backend.Backblaze.B2+<>c__DisplayClass28_0.<PutAsync>b__1 (Duplicati.Library.Utility.AsyncHttpRequest req, System.Threading.CancellationToken reqCancelToken) [0x0008a] in <46bff53630dc43f0ba3250ee17890759>:0 
  at Duplicati.Library.JSONWebHelper.GetJSONDataAsync[T] (System.String url, System.Threading.CancellationToken cancelToken, System.Action`1[T] setup, System.Func`3[T1,T2,TResult] setupbodyreq) [0x000c1] in <b470e1d942864ef99f5a91f39f417fa8>:0 
  at Duplicati.Library.Backend.Backblaze.B2.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x003c7] in <46bff53630dc43f0ba3250ee17890759>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x0052b] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass17_0.<UploadFileAsync>b__0 () [0x0010a] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a3] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00847] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x0089e] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Stream has been closed'.
  at System.IO.FileStream.BeginRead (System.Byte[] array, System.Int32 offset, System.Int32 numBytes, System.AsyncCallback userCallback, System.Object stateObject) [0x0000d] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.Stream+<>c.<BeginEndReadAsync>b__45_0 (System.IO.Stream stream, System.IO.Stream+ReadWriteParameters args, System.AsyncCallback callback, System.Object state) [0x00000] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncTrim[TInstance,TArgs] (TInstance thisRef, TArgs args, System.Func`5[T1,T2,T3,T4,TResult] beginMethod, System.Func`3[T1,T2,TResult] endMethod) [0x00008] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.Stream.BeginEndReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x0001b] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.Stream.ReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count, System.Threading.CancellationToken cancellationToken) [0x00009] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at System.IO.FileStream.ReadAsync (System.Byte[] buffer, System.Int32 offset, System.Int32 count, System.Threading.CancellationToken cancellationToken) [0x00000] in <d2ec5c92492f4d6ba8c422bdf574b786>:0 
  at Duplicati.Library.Utility.Utility.CopyStreamAsync (System.IO.Stream source, System.IO.Stream target, System.Boolean tryRewindSource, System.Threading.CancellationToken cancelToken, System.Byte[] buf) [0x0005e] in <18a4f5320dc54e9fa1cd255415a084f9>:0 
  at Duplicati.Library.Utility.Utility.CopyStreamAsync (System.IO.Stream source, System.IO.Stream target, System.Threading.CancellationToken cancelToken) [0x00077] in <18a4f5320dc54e9fa1cd255415a084f9>:0 
  at Duplicati.Library.Backend.Backblaze.B2+<>c__DisplayClass28_0.<PutAsync>b__1 (Duplicati.Library.Utility.AsyncHttpRequest req, System.Threading.CancellationToken reqCancelToken) [0x0008a] in <46bff53630dc43f0ba3250ee17890759>:0 
  at Duplicati.Library.JSONWebHelper.GetJSONDataAsync[T] (System.String url, System.Threading.CancellationToken cancelToken, System.Action`1[T] setup, System.Func`3[T1,T2,TResult] setupbodyreq) [0x000c1] in <b470e1d942864ef99f5a91f39f417fa8>:0 
  at Duplicati.Library.Backend.Backblaze.B2.PutAsync (System.String remotename, System.IO.Stream stream, System.Threading.CancellationToken cancelToken) [0x003c7] in <46bff53630dc43f0ba3250ee17890759>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Interface.IBackend backend, System.Threading.CancellationToken cancelToken) [0x0052b] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader+<>c__DisplayClass17_0.<UploadFileAsync>b__0 () [0x0010a] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x0017c] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry (System.Func`1[TResult] method, Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x003a3] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFileAsync (Duplicati.Library.Main.Operation.Common.BackendHandler+FileEntryItem item, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x000da] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadBlockAndIndexAsync (Duplicati.Library.Main.Operation.Backup.VolumeUploadRequest upload, Duplicati.Library.Main.Operation.Backup.BackendUploader+Worker worker, System.Threading.CancellationToken cancelToken) [0x00098] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00847] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x0089e] in <e7bec42ecb504e55951d31db4472cf8b>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 <---
<---
<---

Antyhing I am doing wrong ?

drwtsn32 · January 4, 2020, 11:23pm

Couple quick questions…
Has it ever worked for you?
Are there any files in the B2 bucket?

Welcome to the forum!

u2615 · January 4, 2020, 11:32pm

Hi, thx for the welcome

no , it’s the first time I (try to) use duplicati
and therefore, no, I haven’t yet been able to put any file in the bucket (nor have I a local database)

drwtsn32 · January 4, 2020, 11:42pm

Ok, I was just curious if anything succeeded in getting to the bucket. Your problem seems odd, I have used Duplicati successfully with B2 for years now.

Are you certain the application key allows write access to the bucket?

u2615 · January 4, 2020, 11:58pm

Mm fairly sure (capabilities: deleteFiles, listBuckets, listFiles, readFiles, shareFiles, writeFiles), I also tried with the master key…

ts678 · January 5, 2020, 2:46pm

The upload attempts look suspiciously like retries of the same file. This is more obvious in the web UI. Was all of this from a command line (shell prompt) run? That’s less friendly, but may be what you got.

Web UI makes it easier to set up and debug. Is this a command line run, hand-built from the manual? That’s a hard way. Easier is template on web UI (even local), then Export as Command-line, edit, run.

Simple first step to get a better view of what’s going on is to add --console-log-level=Retry to any run.

More work is Duplicati.CommandLine.BackendTester.exe (automatic, given an empty pseudo-folder).

More work, with you calling the shots to see what works, is Duplicati.CommandLine.BackendTool.exe.

Getting a suitable URL varies with the provider. Export (as above) from a web UI for B2 backup gave:

"b2://bucket/pseudo-folder?auth-username=shorter-hex-string&auth-password=longer-hex-string"

u2615 · January 6, 2020, 3:21pm

Thx a lot for your reply, you were quite right: My command line seemed to be malformed. Running Duplicati.CommandLine.BackendTester.exe without problem narrowed the problem to options I set. Although I don’t really know where was the problem (a typo somewhere ?), rewriting everything solved it. Thx all for your time and quick answers, and sorry for such a stupid mistake