Ok, yes, so it seem quite clear that long running (hours) backups are what triggers the issue.
As mentioned above, I do have a couple of pull requests which might improve, hopefully solve, it. Until they are reviewed/merged/tested I cannot spend any more time investigating the issue. One of the pull requests are in the oauth service (#10 ), but the important ones are in the Duplicati codebase, and the first one is the crucial one if the theory is correct:
duplicati:master ← albertony:jottacloud-disable-v2-authid
opened 12:54PM - 23 Jun 22 UTC
Hopefully fixing the authentication problems with Jottacloud after transition to… OAuth.
Assuming OAuth is configured using the oauth service, with a regular (v1) authid. Duplicati then supplies the authid and asks the oauth service for an access token. If the oauth service finds it needs to refresh its stored access token it requests a token refresh from the provider, and in this case the response to Duplicati also includes a v2 authid (see [here](https://github.com/duplicati/oauth-handler/blob/4e58e68bac8a4bd52e8e96b8d186951dde33467a/main.py#L718)). Duplicati will then pick up this and use it for following requests instead of the configured original v1 authid. So there is an "auto-upgrade" to v2 authid mechanism. But next time an instance of the backend object is created in Duplicati, it will use the originally configured v1 authid again, and start from "scratch".
Now the problem with this is that Jottacloud is one of a few providers which generates a new refresh token with every access token refresh and invalidates the old. The procedure above works reasonably well in most cases. But not if the backend object not only performs one token refresh, converting the v1 authid to v2, but also a second token refresh where there is a refresh against the provider using the refresh token embedded in the v2 authid. Then the refresh token still stored in the oauth service for the original v1 authid is outdated and invalidated. Next time a backend object is created, starting from the v1 authid from backend options, the oauth service will end up sending a refresh requst to the provider using an invalidated refresh token. When this happens Jottacloud will mark the token as "stale" and within some expiration time, 12 or 24 hours I think, authentication will fail. Jottacloud will return HTTP 400 Bad Request with details `{"error":"invalid_grant","error_description":"Stale token"}`, which in turn leads to Oauth service returning 500 (Internal Server Error), and Duplicati will fail with "Failed to authorize using the OAuth service: Server error. If the problem persists, try generating a new authid...".
... at least that is the theory! If true, and we are lucky, it solves the authentication problems discussed at the end of the following thread: https://forum.duplicati.com/t/jottacloud-error-401-unauthorized/3929/173. But there may also be other issues, triggering the same error in some cases (e.g. threading issues which I have looked into in another pr), but I think this one should be part of the solution at least.
This should probably be handled in the oauth service, e.g. configuring Jottacloud to be not compatible with v2 and then avoid returning the v2 authid in the response. I will look into that. Still, I think it is not wrong to handle this in Duplicati, if nothing else an additional "safety switch", and also the ability to "beta test" and "hotfix" Duplicati is much easier than the oauth service I think!
- Edit: Done here: https://github.com/duplicati/oauth-handler/pull/10
duplicati:master ← albertony:jottacloud-username-from-token
opened 11:42AM - 21 Jun 22 UTC
Refactoring and small performance improvement. See commit messages for details.
…
This was done after debugging on the issue we have with the new oauth authentication in the Jottacloud backend (stale token in oauth service). These changes reduces the number of requests to the oauth service, hence the frequency of usage of the access token. If the stale token is caused by some race condition, perhaps this may have a positive effect, although I don't really have high expectations... I think the change is an improvement regardless.
duplicati:master ← albertony:jottacloud-v2-authid
opened 10:15AM - 23 Jun 22 UTC
Follow up from #7:
1. First commit adds support for v2 authid with cli tokens, … which was simply skipped in #7.
2. Second commit disables v2 authid for providers that uses refresh token rotation, since these are not (currently) compatible, if my understanding is correct! This is related to https://github.com/duplicati/duplicati/pull/4779 which prevents "auto-upgrade" from v1 to v2 authid in Jottacloud backend of Duplicati. The current change in oauth handler would also do the same, but also prevents explicit use of v2 authids, and is a more "consistent" solution. For more background, see https://github.com/duplicati/duplicati/pull/4779.
3. Third commit corrects (hopefully) the "fetch token" handling when using cli token, enabling automatic insert of the authid back into duplicati. (This fetch token thing was something I had a question about in the original pr, but never got an answer).
3 Likes