GDrive - Backup failed, Invalid status code: 308

mango1402 · October 27, 2020, 10:51am

Hi.
I am using Duplicati - 2.0.5.1_beta_2020-01-18 on a Raspi 4.
Backups were running without any problems since I installed in June 2020 weekly directly to Google Drive.
But since about 2 weeks, I only get error messages and nothing is uploaded to GDrive - but there is enough space left and dayli quota (750GB) is definitly not exceeded.

I searched this forum and all over the WWW, but found no similar troubles like mine.
Really don’t know what’s going wrong…

Maybe some of you can help me.

Many thanks in advance,
Manfred

Log:

System.AggregateException: One or more errors occurred. (The channel "BackendRequests" is retired (The channel "BackendRequests" is retired) (One or more errors occurred. (Invalid status code: 308))) ---> System.AggregateException: The channel "BackendRequests" is retired (The channel "BackendRequests" is retired) (One or more errors occurred. (Invalid status code: 308)) ---> CoCoL.RetiredException: The channel "BackendRequests" is retired
  at CoCoL.Channel`1[T].WriteAsync (T value, CoCoL.ITwoPhaseOffer offer) [0x0016b] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x0009d] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00ca8] in <8f1de655bd1240739a78684d845cecc8>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <8f1de655bd1240739a78684d845cecc8>:0 
   --- End of inner exception stack trace ---
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x0005d] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00074] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00349] in <c5f097a49c0a4f1fb0f93cf3f5f218b1>:0 
---> (Inner Exception #0) System.AggregateException: The channel "BackendRequests" is retired (The channel "BackendRequests" is retired) (One or more errors occurred. (Invalid status code: 308)) ---> CoCoL.RetiredException: The channel "BackendRequests" is retired
  at CoCoL.Channel`1[T].WriteAsync (T value, CoCoL.ITwoPhaseOffer offer) [0x0016b] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x0009d] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00ca8] in <8f1de655bd1240739a78684d845cecc8>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <8f1de655bd1240739a78684d845cecc8>:0 
---> (Inner Exception #0) CoCoL.RetiredException: The channel "BackendRequests" is retired
  at CoCoL.Channel`1[T].WriteAsync (T value, CoCoL.ITwoPhaseOffer offer) [0x0016b] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend (Duplicati.Library.Main.BackupResults result, CoCoL.IWriteChannel`1[T] uploadtarget, System.Threading.Tasks.Task uploader) [0x0009d] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00ca8] in <8f1de655bd1240739a78684d845cecc8>:0 <---

---> (Inner Exception #1) System.AggregateException: One or more errors occurred. (Invalid status code: 308) ---> System.Net.ProtocolViolationException: Invalid status code: 308
  at System.Net.HttpWebRequest.Redirect (System.Net.HttpStatusCode code, System.Net.WebResponse response) [0x000e2] in <3d97dda773a54430b892e1d794e5d921>:0 
  at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.HttpWebResponse response) [0x0016f] in <3d97dda773a54430b892e1d794e5d921>:0 
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x0006f] in <3d97dda773a54430b892e1d794e5d921>:0 
  at System.Net.HttpWebRequest.MyGetResponseAsync (System.Threading.CancellationToken cancellationToken) [0x002fd] in <3d97dda773a54430b892e1d794e5d921>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00673] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x0089e] in <8f1de655bd1240739a78684d845cecc8>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Net.ProtocolViolationException: Invalid status code: 308
  at System.Net.HttpWebRequest.Redirect (System.Net.HttpStatusCode code, System.Net.WebResponse response) [0x000e2] in <3d97dda773a54430b892e1d794e5d921>:0 
  at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.HttpWebResponse response) [0x0016f] in <3d97dda773a54430b892e1d794e5d921>:0 
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x0006f] in <3d97dda773a54430b892e1d794e5d921>:0 
  at System.Net.HttpWebRequest.MyGetResponseAsync (System.Threading.CancellationToken cancellationToken) [0x002fd] in <3d97dda773a54430b892e1d794e5d921>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00673] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x0089e] in <8f1de655bd1240739a78684d845cecc8>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 <---
<---
<---

ts678 · October 30, 2020, 11:05pm

Bug 28287 Summary: WebException.Response is null under Mono, not null under .net appears similar.

308 Permanent Redirect explains how Google uses 308 in a non-standard way as Resume Incomplete

You can try researching that, but I wonder if your upload stalled due to some sort of a network problem.

Have you raised dblock-size (a.k.a Remote Volume size on screen 5 Options beyond default 50 MB?

You can watch About → Show log → Live → Retry to see files try to upload while watching Pi network traffic using something (I assume). Ideally traffic should happen when upload starts, and go until done…

EDIT:

From your log, it looks like a 308 came back from an upload and mono wound up considering it an error:

—> (Inner Exception #1) System.AggregateException: One or more errors occurred. (Invalid status code: 308) —> System.Net.ProtocolViolationException: Invalid status code: 308
at System.Net.HttpWebRequest.Redirect (System.Net.HttpStatusCode code, System.Net.WebResponse response) [0x000e2] in <3d97dda773a54430b892e1d794e5d921>:0
at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.HttpWebResponse response) [0x0016f] in <3d97dda773a54430b892e1d794e5d921>:0
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x0006f] in <3d97dda773a54430b892e1d794e5d921>:0
at System.Net.HttpWebRequest.MyGetResponseAsync (System.Threading.CancellationToken cancellationToken) [0x002fd] in <3d97dda773a54430b892e1d794e5d921>:0
at Duplicati.Library.Main.Operation.Backup.BackendUploader.b__13_0 (<>f__AnonymousType12`1[j__TPar] self) [0x00673] in <8f1de655bd1240739a78684d845cecc8>:0

I’d note that the mono bug report at its end does give a workaround that I don’t think Duplicati is using.

308 appears to be a longstanding sore spot, and others have worked on it. Here’s a comment in code:

github.com

duplicati/duplicati/blob/b6e16189476648c7d260fb8ade9cc63599c9fa68/Duplicati/Library/Backend/GoogleServices/GoogleCommon.cs#L64-L76


      
          if (code == 308)

          {

              // A lack of a Range header is undocumented, 

              // but seems to occur when no data has reached the server:

              // https://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3884

          

              if (resp.Headers["Range"] == null)

                  return 0; 

              else

                  return long.Parse(resp.Headers["Range"].Split(new char[] { '-' })[1]) + 1;

          }

          else

              throw new WebException(string.Format("Unexpected status code: {0}", code), null, WebExceptionStatus.ServerProtocolViolation, resp);

mango1402 · November 2, 2020, 7:57am

Hi,
Thanks for your researches and comments.
In the meantime I tried something:

same installation on Raspi, but use MS One drive -> failure
same installation on Raspi, but use dropbox -> failure
installation of Ubuntu as Virtual machine on Win10 PC -> same failures
compleatly new installation on Raspi -> same failures as befoe
updated to duplicati 2.0.5.111-1 -> same as before
Backup of single text file (~2kByte) -> OK
Backup of 8MB ZIP file -> not OK
change Block size to 1024kByte -> not OK
installed Duplicati on Win10 -> OK!!??!!??

looks like it is not a (private) Network problem. More or less a Problem on linux installallations, maybe mono?

The strange is: up to October, 21 ALL Backups from my Raspi installations worked verry well each week. But then 3 Tasks were completed, and one started to stuck with mentioned error.
There was no update of any components of the System running since June…

ts678 · November 2, 2020, 4:44pm

Note the blocksize is completely different than dblock-size which I mentioned earlier. Which did you set? Please describe this and other results better if possible. I can’t tell if each one is 308 (odd) or other error.
Seeing 308 from every destination type (and some weren’t given) conflicts with an odd-Google-use idea, however I don’t know what “failure” or “not OK” reports . I’d guess “same * before” means a 308. Clarify? Where a different destination type was not noted, it correlates with “same”. So others were other errors?