Upgrading GAE OAuth Server

kenkendk · January 3, 2025, 2:53pm

On January 13th 2025 we will update the OAuth service running in Google App Engine.

The current version is running a deprecated Python 2.7 and the upgrade is going to be with Python 3. We have tested as much as possible that this update will not cause any problems. Unfortunately, since Python 2.7 is deprecated, we will not be able to roll back the changes.

Update: As of 2025-01-14 10:00 UTC service is again running and appears stable. We continue to monitor the service.

Long-term, the plan is to migrate away from AppEngine and run a hosted version of the OAuth server. Before we can make that switch smoother, we need to have the current instance running in a way where we can update it, which is not possible now.

If you would prefer to run a self-hosted version of the OAuth server, this is possible with the OAuth server source code or using the pre-built OAuth server Docker image.

Affected backends:

box.com
Dropbox
Google Cloud Storage
Google Drive
JottaCloud
SharePoint v2
OneDrive
Microsoft Groups

If you would like to review the upgrade before it is applied, the OAuth server PR is ready:

github.com/duplicati/oauth-handler

Migrate to Python 3 App Engine

duplicati:master ← carljohnsen:master

opened 05:45PM - 20 Dec 24 UTC

carljohnsen

+881 -866

This pull request migrates to Python 3. Most notable changes: - The cryptog…raphy package in Python 2, the now unmaintained [`pycrypto`](https://pypi.org/project/pycrypto/), has been replaced with the maintained drop-in replacement fork [`pycryptodome`](https://pypi.org/project/pycryptodome/). - The app engine db has been replaced by [`google.cloud.ndb`](https://pypi.org/project/google-cloud-ndb/) - Memcache is still being used, but through googles compatibility layer [`appengine-python-standard'](https://pypi.org/project/appengine-python-standard/) This has been tested with test users on Box, Dropbox, Google and Jottacloud.

kenkendk · January 13, 2025, 12:36pm

Upgrade is ongoing, most requests seems to pass.
A few requests are failing, we are investigating.

kenkendk · January 13, 2025, 2:54pm

There are some performance issues with the service currently.
We are investigating.

Hannes3 · January 13, 2025, 3:37pm

Is version 2.1.0.106_canary using the new OAuth?
My back-up to OneDrive failed several times.
After several restarts it is working fine!

kenkendk · January 13, 2025, 4:17pm

Yes, all versions of Duplicati is using the same OAuth on Google App Engine, unless you specify a different server.

There were a few periods of 500 errors due to scaling issues.
It looks like it is mostly now, but using 10x the resources from before.

DCinME · January 14, 2025, 1:51am

As you indicated, one of my backup jobs this evening failed with a 500 error. Re-ran the job and it worked fine the second time.

kenkendk · January 14, 2025, 10:40am

Service is now running stable having requests served with less than 100ms average response time.

It was a scaling issue caused by an extremely slow fallback in PyCrypto causing requests to take up to 8 seconds to complete.

During peak times this would exhaust all resources and return a 500 error to unlucky callers. Now the issue is fixed and resource usage is back to what it was before the upgrade and there is ample room to scale in case of large request peaks.