Jottacloud Error 401 (Unauthorized)

shalmirane · June 19, 2022, 4:22pm

Made myself a little imacro to replace all tokens via loop - you need to update the “your token there” value (and perhaps URL), as well to launch via loop 1…X, knowing how many of the configs you got (10 in my case).

URL GOTO=http://localhost:8200/ngax/index.html#/
SET !VAR1 "Your token there"

TAG POS={{!LOOP}} TYPE=SPAN ATTR=TXT:Edit<SP>…
TAG POS=1 TYPE=SPAN ATTR=TXT:2
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:oauth_authid CONTENT={{!VAR1}}
TAG POS=1 TYPE=SPAN ATTR=TXT:5
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=ID:save
WAIT SECONDS=1

If anybody knows how to…

Detect number of backups on page and make loop around that code without needing to count them yourself
Prompt user for token (making the update of script unnecessary) once, before the first loop

albertony · June 19, 2022, 6:03pm

I’m curious: Does everyone have problems like this:

Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.

Or does it just work for some of you as well? I only have a small backup set up myself, but have not seen this issue yet…

shalmirane · June 19, 2022, 6:08pm

Pretty much everybody See the logs above - any longer operation will lead to the token not being refreshed correctly and that operation will fail as well as any other afterward.

When I was pushing backups for about 1TB at 3MB/s (because of the same error I was not able to repair that DB - it takes too much time and it expired every time; unlike repair, upload can be continued from where it stopped and for this particular backup, I did not need history that much).

It crashed like every 12 hours. I had to regenerate the token and replace it in all backup settings. Having 10 of them, that’s why the macro above to replace it in all configs. Native support for that would be cooler though.

waLIEN · June 20, 2022, 6:10am

I’m getting this error consistently. Should you need more logs or if you want to do a remote session, I’d be more than willing to help out as much as I can.

albertony · June 20, 2022, 8:36am

Not sure, but I think logs from the oauth service are needed to get relevant info, so that is a bit of a challenge. I will try some more to reproduce it locally, maybe with --throttle-upload or something, to make the backup operation slower (or is there any other tricks you can think of?). Token renewing per se seem to work, as even with a small backup it happens every time the backup is run when it is more than one hour since the token stored in oauth service was last refreshed. So I assume the problem is related to backups (or repairs) taking more than one hour, but seems not consistently related to just that one hour refresh either. Tried to time my backup runs to make a token refresh required in the middle of it, but did not get any errors out of that.

FreaQ · June 20, 2022, 2:31pm

I can confirm this. Approx every 12h the token gets invalid.
If it is due to heavy backup operations or sth. else I cannot tell.

waLIEN · June 20, 2022, 9:23pm

I just had an OAUTH token expire in <6 hours:

Failed: One or more errors occurred.
Details: System.AggregateException: One or more errors occurred. ---> System.AggregateException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> Duplicati.Library.Interface.UserInformationException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at Duplicati.Library.Utility.AsyncHttpRequest.AsyncWrapper.GetResponseOrStream()
   at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse()
   at Duplicati.Library.JSONWebHelper.GetResponse(AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T](AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.OAuthHelper.GetTokenResponse[T]()
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   --- End of inner exception stack trace ---
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   at Duplicati.Library.OAuthHelper.CreateRequest(String url, String method, Boolean noAuthorization)
   at Duplicati.Library.JSONWebHelper.GetJSONData[T](String url, Action`1 setup, Action`1 setupbodyreq)
   at Duplicati.Library.Backend.JottacloudAuthHelper..ctor(String accessToken)
   at Duplicati.Library.Backend.Jottacloud..ctor(String url, Dictionary`2 options)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.DynamicLoader.BackendLoader.BackendLoaderSub.GetBackend(String url, Dictionary`2 options)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>b__20_0()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.GetWorker(List`1 workers)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFilesets(List`1 workers, List`1 filesets)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
   --- End of inner exception stack trace ---
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
   --- End of inner exception stack trace ---
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
---> (Inner Exception #0) System.AggregateException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> Duplicati.Library.Interface.UserInformationException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at Duplicati.Library.Utility.AsyncHttpRequest.AsyncWrapper.GetResponseOrStream()
   at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse()
   at Duplicati.Library.JSONWebHelper.GetResponse(AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T](AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.OAuthHelper.GetTokenResponse[T]()
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   --- End of inner exception stack trace ---
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   at Duplicati.Library.OAuthHelper.CreateRequest(String url, String method, Boolean noAuthorization)
   at Duplicati.Library.JSONWebHelper.GetJSONData[T](String url, Action`1 setup, Action`1 setupbodyreq)
   at Duplicati.Library.Backend.JottacloudAuthHelper..ctor(String accessToken)
   at Duplicati.Library.Backend.Jottacloud..ctor(String url, Dictionary`2 options)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.DynamicLoader.BackendLoader.BackendLoaderSub.GetBackend(String url, Dictionary`2 options)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>b__20_0()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.GetWorker(List`1 workers)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFilesets(List`1 workers, List`1 filesets)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
   --- End of inner exception stack trace ---
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
---> (Inner Exception #0) Duplicati.Library.Interface.UserInformationException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at Duplicati.Library.Utility.AsyncHttpRequest.AsyncWrapper.GetResponseOrStream()
   at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse()
   at Duplicati.Library.JSONWebHelper.GetResponse(AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T](AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.OAuthHelper.GetTokenResponse[T]()
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   --- End of inner exception stack trace ---
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   at Duplicati.Library.OAuthHelper.CreateRequest(String url, String method, Boolean noAuthorization)
   at Duplicati.Library.JSONWebHelper.GetJSONData[T](String url, Action`1 setup, Action`1 setupbodyreq)
   at Duplicati.Library.Backend.JottacloudAuthHelper..ctor(String accessToken)
   at Duplicati.Library.Backend.Jottacloud..ctor(String url, Dictionary`2 options)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.DynamicLoader.BackendLoader.BackendLoaderSub.GetBackend(String url, Dictionary`2 options)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>b__20_0()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.GetWorker(List`1 workers)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFilesets(List`1 workers, List`1 filesets)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()<---

---> (Inner Exception #1) System.AggregateException: One or more errors occurred. ---> Duplicati.Library.Interface.UserInformationException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at Duplicati.Library.Utility.AsyncHttpRequest.AsyncWrapper.GetResponseOrStream()
   at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse()
   at Duplicati.Library.JSONWebHelper.GetResponse(AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T](AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.OAuthHelper.GetTokenResponse[T]()
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   --- End of inner exception stack trace ---
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   at Duplicati.Library.OAuthHelper.CreateRequest(String url, String method, Boolean noAuthorization)
   at Duplicati.Library.JSONWebHelper.GetJSONData[T](String url, Action`1 setup, Action`1 setupbodyreq)
   at Duplicati.Library.Backend.JottacloudAuthHelper..ctor(String accessToken)
   at Duplicati.Library.Backend.Jottacloud..ctor(String url, Dictionary`2 options)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.DynamicLoader.BackendLoader.BackendLoaderSub.GetBackend(String url, Dictionary`2 options)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>b__20_0()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.GetWorker(List`1 workers)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFilesets(List`1 workers, List`1 filesets)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) Duplicati.Library.Interface.UserInformationException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at Duplicati.Library.Utility.AsyncHttpRequest.AsyncWrapper.GetResponseOrStream()
   at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse()
   at Duplicati.Library.JSONWebHelper.GetResponse(AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T](AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.OAuthHelper.GetTokenResponse[T]()
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   --- End of inner exception stack trace ---
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   at Duplicati.Library.OAuthHelper.CreateRequest(String url, String method, Boolean noAuthorization)
   at Duplicati.Library.JSONWebHelper.GetJSONData[T](String url, Action`1 setup, Action`1 setupbodyreq)
   at Duplicati.Library.Backend.JottacloudAuthHelper..ctor(String accessToken)
   at Duplicati.Library.Backend.Jottacloud..ctor(String url, Dictionary`2 options)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.DynamicLoader.BackendLoader.BackendLoaderSub.GetBackend(String url, Dictionary`2 options)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>b__20_0()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.GetWorker(List`1 workers)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFilesets(List`1 workers, List`1 filesets)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()<---
<---
<---


Log data:
2022-06-20 14:11:25 -07 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler.appspot.com?type=jottacloud ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at Duplicati.Library.Utility.AsyncHttpRequest.AsyncWrapper.GetResponseOrStream()
   at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse()
   at Duplicati.Library.JSONWebHelper.GetResponse(AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.JSONWebHelper.ReadJSONResponse[T](AsyncHttpRequest req, Object requestdata)
   at Duplicati.Library.OAuthHelper.GetTokenResponse[T]()
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   --- End of inner exception stack trace ---
   at Duplicati.Library.OAuthHelper.get_AccessToken()
   at Duplicati.Library.OAuthHelper.CreateRequest(String url, String method, Boolean noAuthorization)
   at Duplicati.Library.JSONWebHelper.GetJSONData[T](String url, Action`1 setup, Action`1 setupbodyreq)
   at Duplicati.Library.Backend.JottacloudAuthHelper..ctor(String accessToken)
   at Duplicati.Library.Backend.Jottacloud..ctor(String url, Dictionary`2 options)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.DynamicLoader.BackendLoader.BackendLoaderSub.GetBackend(String url, Dictionary`2 options)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>b__20_0()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.GetWorker(List`1 workers)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.UploadFilesets(List`1 workers, List`1 filesets)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()

kenkendk · June 21, 2022, 5:35pm

There is nothing in the logs really. All the jottacloud refresh errors I can see are the same:

It just says “Stale token” indicating that the token has expired for some reason.

More alarmingly, I see some similar errors for the other backends, which could indicate that the refresh token is not persisted after being refreshed, so the old refresh token is re-used, causing a login problem later. If this is the case, anyone affected by it would need to obtain a new auth-token .

Update: it appears it was a cluster of bad requests in the logs, there are many that are working and no flood of user reports indicates that the service is working as before for non-Jottacloud entries.

waLIEN · June 21, 2022, 6:29pm

I generated a new OAUTH token ~22 hours ago. So far, it’s been working. I’m not sure what changed, but this is the longest one token has lasted. I’ll keep you guys updated when/if this token expires.

FreaQ · June 22, 2022, 5:56pm

Was running a backup for over 24hours without a problem.
I got OAuth error when three dlist and dindex files were uploaded within a minute.
I assume the token gets stale if to many refresh request come in within a defined period.
@kenkendk

w.geronius · June 23, 2022, 7:09am

Since I started updating my configs ~ 10 days ago (wih no extra parms), I have not seen this issue so far… still using the same authids.

albertony · June 23, 2022, 2:23pm

Real head-scratcher this one…

But I do have a pull request which might have a positive effect: Disable automatic use of v2 authid for Jottacloud by albertony · Pull Request #4779 · duplicati/duplicati · GitHub

It would be interesting to get this tested on a reproducable case. I don’t remember if anyone of you experiencing problems were able to build Duplicati from source? Or maybe someone have the super-powers to fast-track a review+merge+deploy…

aestetix · June 23, 2022, 3:11pm

I’m still using rclone and not having any issues, so whatever rclone does seems to work.

albertony · June 23, 2022, 4:10pm

I’m using rclone with Jottacloud myself, but separately - without Duplicati. I’m a contributor to the project and familiar with its implementation, though not all the inner details of the OAuth handling. Anyway, its not really comparable to the Duplicati implementation. Duplicati have the complicating factor of OAuth handling being in a separate web service; more moving parts, different language (Python 2), Google App Engine deploy/dependencies, difficult to test and debug… And then the time aspect of this problem does not make it easier (the fact it takes a large backup running for a long time, then a wait 12-24 hours, before the error surfaces).

waLIEN · June 24, 2022, 5:12pm

I’m a few days in now – the backups are still completing successfully 3 days after I regenerated an OAUTH key.

shalmirane · June 26, 2022, 2:01pm

It fails for me once I run a backup on the folder with Acronis backup of my C drive - dozen GB’s that take hours over restricted jottacloud upload speed + like two hours long deletion of old backups (every time, even if there’s nothing to backup, e.g. when I rerun backup immediately after it completes). So basically every Saturday

I do suspect that other scheduled backups, that fit in the time this one is running, somehow interfere with it.

albertony · June 26, 2022, 2:12pm

Thank you for the update.

I suspect it has something to do with the length of time a backup is running. So if other backups are running the Duplicati backup is slower, and that increases the probability of this token problem. Several hours of running time is “high risk” because token refresh happens every hour, and it is the token refreshing that triggers it, perhaps it needs to be multiple token refreshes in the same run. At least that is one theory.

In very large files, which takes hours to process?

shalmirane · June 26, 2022, 2:29pm

This Saturday backup is 23 files of 173GB (to process, actual uploaded blocks were around 80GB iirc, when duplicati throwed an error for the first time on Saturday). Duplicati reports upload speed of 3MB/sec (I’m at 7TB storage, two above the 5TB “unlimited” tier). That would be 7,5 hours of pure upload if I go with that 80GB, but I also know it was not at full 3MB/sec for the whole time, so might be a lot more (I had it every 12 hours when doing initial re-upload of Dropbox backups).

Today when I re-run the backup, for it to complete, it was cut in middle by other scheduled backups that run daily (e.g. the dropbox; Acronis folder one runs weekly, a day after Acronis actually does the snapshot) - my guess would be after one of those Acronis folder *.tibs files completed upload. So this might be about an hour later after I regenerated the token after yesterday’s failure.

If it’s possible to fetch from logs, I could try to provide more accurate numbers, but I’m not really lucky with how duplicati handles logging on so many places with some viewable only when the backup is happening, others not displaying errors even when some are thrown; my UX with duplicati way of logs is very bad.

FreaQ · June 27, 2022, 8:41am

Another two long running (>24h) failed with OAuth.

albertony · June 27, 2022, 8:55am

Ok, yes, so it seem quite clear that long running (hours) backups are what triggers the issue.

As mentioned above, I do have a couple of pull requests which might improve, hopefully solve, it. Until they are reviewed/merged/tested I cannot spend any more time investigating the issue. One of the pull requests are in the oauth service (#10), but the important ones are in the Duplicati codebase, and the first one is the crucial one if the theory is correct:

github.com/duplicati/duplicati

Disable automatic use of v2 authid for Jottacloud

duplicati:master ← albertony:jottacloud-disable-v2-authid

opened 12:54PM - 23 Jun 22 UTC

albertony

+12 -1

Hopefully fixing the authentication problems with Jottacloud after transition to… OAuth. Assuming OAuth is configured using the oauth service, with a regular (v1) authid. Duplicati then supplies the authid and asks the oauth service for an access token. If the oauth service finds it needs to refresh its stored access token it requests a token refresh from the provider, and in this case the response to Duplicati also includes a v2 authid (see [here](https://github.com/duplicati/oauth-handler/blob/4e58e68bac8a4bd52e8e96b8d186951dde33467a/main.py#L718)). Duplicati will then pick up this and use it for following requests instead of the configured original v1 authid. So there is an "auto-upgrade" to v2 authid mechanism. But next time an instance of the backend object is created in Duplicati, it will use the originally configured v1 authid again, and start from "scratch". Now the problem with this is that Jottacloud is one of a few providers which generates a new refresh token with every access token refresh and invalidates the old. The procedure above works reasonably well in most cases. But not if the backend object not only performs one token refresh, converting the v1 authid to v2, but also a second token refresh where there is a refresh against the provider using the refresh token embedded in the v2 authid. Then the refresh token still stored in the oauth service for the original v1 authid is outdated and invalidated. Next time a backend object is created, starting from the v1 authid from backend options, the oauth service will end up sending a refresh requst to the provider using an invalidated refresh token. When this happens Jottacloud will mark the token as "stale" and within some expiration time, 12 or 24 hours I think, authentication will fail. Jottacloud will return HTTP 400 Bad Request with details `{"error":"invalid_grant","error_description":"Stale token"}`, which in turn leads to Oauth service returning 500 (Internal Server Error), and Duplicati will fail with "Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid...". ... at least that is the theory! If true, and we are lucky, it solves the authentication problems discussed at the end of the following thread: https://forum.duplicati.com/t/jottacloud-error-401-unauthorized/3929/173. But there may also be other issues, triggering the same error in some cases (e.g. threading issues which I have looked into in another pr), but I think this one should be part of the solution at least. This should probably be handled in the oauth service, e.g. configuring Jottacloud to be not compatible with v2 and then avoid returning the v2 authid in the response. I will look into that. Still, I think it is not wrong to handle this in Duplicati, if nothing else an additional "safety switch", and also the ability to "beta test" and "hotfix" Duplicati is much easier than the oauth service I think! - Edit: Done here: https://github.com/duplicati/oauth-handler/pull/10

github.com/duplicati/duplicati

Jottacloud use username from token

duplicati:master ← albertony:jottacloud-username-from-token

opened 11:42AM - 21 Jun 22 UTC

albertony

+114 -78

Refactoring and small performance improvement. See commit messages for details. … This was done after debugging on the issue we have with the new oauth authentication in the Jottacloud backend (stale token in oauth service). These changes reduces the number of requests to the oauth service, hence the frequency of usage of the access token. If the stale token is caused by some race condition, perhaps this may have a positive effect, although I don't really have high expectations... I think the change is an improvement regardless.

github.com/duplicati/oauth-handler

CLI Token / Jottacloud follow up

duplicati:master ← albertony:jottacloud-v2-authid

opened 10:15AM - 23 Jun 22 UTC

albertony

+58 -12

Follow up from #7: 1. First commit adds support for v2 authid with cli tokens, …which was simply skipped in #7. 2. Second commit disables v2 authid for providers that uses refresh token rotation, since these are not (currently) compatible, if my understanding is correct! This is related to https://github.com/duplicati/duplicati/pull/4779 which prevents "auto-upgrade" from v1 to v2 authid in Jottacloud backend of Duplicati. The current change in oauth handler would also do the same, but also prevents explicit use of v2 authids, and is a more "consistent" solution. For more background, see https://github.com/duplicati/duplicati/pull/4779. 3. Third commit corrects (hopefully) the "fetch token" handling when using cli token, enabling automatic insert of the authid back into duplicati. (This fetch token thing was something I had a question about in the original pr, but never got an answer).