Jottacloud Error 401 (Unauthorized)

ning42 · June 5, 2022, 6:00am

I switched to duplicati and jotta 3 months ago and used the built-in jottacloud connector. But with this change, I’m guessing I have to move to rclone as well.

I’m using the linuxserver/duplicati docker image on a QNAP NAS. And not 100% sure how to get rclone to work - is there by chance anyone who can give a tip on how to get the rclone part configured - it looks like the gui need a rclone config to be working “underneath”

Any tips would be appreciated

shalmirane · June 5, 2022, 1:12pm

Once the patch is vetted and makes it to the main branch, we should be fine.

waLIEN · June 6, 2022, 5:58pm

Thanks for keeping us updated on this.

shalmirane · June 6, 2022, 6:00pm

Oh, I’m not a developer, just watching Jottacloud removed basic authentication, Duplicati can no longer connect · Issue #4697 · duplicati/duplicati · GitHub and hoping it gets published soon, as I’m not able to even launch that dev environment, let alone compile something I could actually use to backup.

So actually without backups being able neither to backup, nor to restore them.

ning42 · June 7, 2022, 4:51am

Success !!

Used the skaro13/duplicati-linuxserver:v2.0.6.3-2.0.6.3_beta_2021-06-17-ls135_pr-4699 as mentioned in Jottacloud removed basic authentication, Duplicati can no longer connect · Issue #4697 · duplicati/duplicati · GitHub

The instructions described worked perfectly - and backup is now running

teknolai · June 7, 2022, 10:22am

I see that you got jottacloud running with the patch, but in case you would like to run it with rclone instead, I thought I’d give some tips on how I did that. I’m did this on Windows:

Download and unzip rclone. Add the rclone.exe to your Windows PATH (Environment variables).
In case you are running duplicati as a service, I believe it is a good idea to make sure the rclone config ends up in a commonly available folder (and not your user folder - this was the default in my case).
Configure Rclone with both a Jottacloud remote (I used this guide: Jottacloud) AND a “local” remote (this means a rclone storage type local). This is described here: Jottacloud Error 401 (Unauthorized) - #101 by aestetix
Either create a new backup, or change destination inan existing backup

After testing this on a test-backup, I replaced the jottacloud destination with jottacloud-through-rclone destination without any issues on all my live backups. It worked like a charm!

FreaQ · June 7, 2022, 12:41pm

Where did you find this docker image?
On the Docker hub it is not listed and on the github it is not linked.

I would like to test it.

ning42 · June 7, 2022, 1:35pm

I followed Jottacloud removed basic authentication, Duplicati can no longer connect · Issue #4697 · duplicati/duplicati · GitHub, i think it is private that is why you can’t find it, But can ask

shalmirane · June 8, 2022, 7:00pm

docker pull skaro13/duplicati-linuxserver:v2.0.6.3-2.0.6.3_beta_2021-06-17-ls135_pr-4699

FreaQ · June 9, 2022, 6:18am

I can confirm that the PR #4699 is working.
Using it with the docker image from skaro13.

FreaQ · June 9, 2022, 10:33am

Got an error on the OAuth beta server while trying to repair a database.

Failed to dispose backend instance: Failed to authorize using the OAuth service: Too many request for this key, wait 60 minutes. If the problem persists, try generating a new authid token from: https://duplicati-oauth-handler-beta.appspot.com?type=jottacloud

Is this error related to limitations to the OAuth beta server or Jottacloud? And if it is a limitation to the beta server are there the same limitations to the production server?

albertony · June 9, 2022, 10:52am

The error message comes from the oauth service:

github.com

duplicati/oauth-handler/blob/afc902bb816ebc9374f87ea26ad4289729f3839d/main.py#L514


      
          
          if settings.RATE_LIMIT > 0:
          
              ratelimiturl = '/ratelimit?id=' + keyid + '&adr=' + self.request.remote_addr
              ratelimit = memcache.get(ratelimiturl)
          
              if ratelimit is None:
                  memcache.add(key=ratelimiturl, value=1, time=60 * 60)
              elif ratelimit > settings.RATE_LIMIT:
                  logging.info('Rate limit response to: %s', keyid)
                  self.response.headers['X-Reason'] = 'Too many request for this key, wait 60 minutes'
                  self.response.set_status(503, 'Too many request for this key, wait 60 minutes')
                  return
              else:
                  memcache.incr(ratelimiturl)
          
          cacheurl = '/refresh?id=' + keyid + '&h=' + hashlib.sha256(password).hexdigest()
          
          cached_res = memcache.get(cacheurl)
          if cached_res is not None and type(cached_res) != type(''):
              exp_secs = (int)((cached_res['expires'] - datetime.datetime.utcnow()).total_seconds())

The rate limit feature can be enabled or disabled, default disabled:

github.com

duplicati/oauth-handler/blob/afc902bb816ebc9374f87ea26ad4289729f3839d/settings.py#L39


      
          GD_CLIENT_SECRET = 'XXXXXXXXXXXXXXXXXXXX'
          HC_CLIENT_ID = 'XXXXXXXXXXXXXXXXXXXX'
          HC_CLIENT_SECRET = 'XXXXXXXXXXXXXXXXXXXX'
          AMZ_CLIENT_ID = 'XXXXXXXXXXXXXXXXXXXX'
          AMZ_CLIENT_SECRET = 'XXXXXXXXXXXXXXXXXXXX'
          BOX_CLIENT_ID = 'XXXXXXXXXXXXXXXXXXXX'
          BOX_CLIENT_SECRET = 'XXXXXXXXXXXXXXXXXXXX'
          DROPBOX_CLIENT_ID = 'XXXXXXXXXXXXXXXXXXXX'
          DROPBOX_CLIENT_SECRET = 'XXXXXXXXXXXXXXXXXXXX'
          
          RATE_LIMIT = 0
          
          try:
              from config import WL_CLIENT_ID
              from config import WL_CLIENT_SECRET
          except ImportError:
              pass
          
          try:
              from config import MSGRAPH_CLIENT_ID
              from config import MSGRAPH_CLIENT_SECRET

The configuration template “suggests” enabling it with a value of 4 (requests per hour):

github.com

duplicati/oauth-handler/blob/582bb95ddf9e3413da64e4e59a7212cf26e1e469/config-template.py#L19


      
          # APP_NAME = 'CHANGE-ME-IN-CONFIG.PY'
          
          # Name to display on front page
          # SERVICE_DISPLAYNAME = APP_NAME + ' OAuth Handler'
          
          # Callback URI for OAuth
          # OAUTH_CALLBACK_URI = 'https://' + APP_DOMAIN + '/logged-in'
          
          # Sets a limit for how many requests can be performed in
          # an hour for a single keyid+ip pair
          # RATE_LIMIT = 4
          
          # Conditional setup example:
          # if TESTING:
          #     OAUTH_CALLBACK_URI = 'http://localhost:12080/logged-in'
          
          # If you need to modify the lookup table and service list,
          # add this method
          # def POST_CONFIG(lookup, services):
          #     pass

It was obviously enabled on the beta service instance (perhaps with the value 4 from the template), but I have no idea if it is enabled or not in the prod service. I have seen the message myself when testing the beta service, but have no prior experience with the oauth service so don’t know how the prod instance behaves.

albertony · June 12, 2022, 9:37pm

Great news: Support for Jottacloud OAuth (via CLI token) is “official” .

The official production version of the OAuth handler web service has been updated (https://duplicati-oauth-handler.appspot.com/).
Duplicati has been updated, released as new canary (v2.0.6.103-2.0.6.103_canary_2022-06-12).

Big thanks to @jthall for your valuable input, and for your effort with the beta oauth service!

Please test with latest canary and report any issues. Not all of the steps described previously will now be necessary, but something like this:

Visit https://jottacloud.com/web/secure and log in. Click “Generate” in the “Personal login token” section, and enter your password again. Copy the resultant CLI token.
Open Duplicati, and edit a backup or create a new backup.
Click the “AuthID” link to open a “pop-up” with the official OAuth service showing Jottacloud only. Alternatively just open the main page of the oauth service at https://duplicati-oauth-handler.appspot.com in another tab. Click “Jottacloud login”, paste in the token obtained in the previous step and click “Login”. Copy the resultant AuthID.
Paste in the AuthID you obtained previously in the “AuthID” field
Click test connection and if successful, click through with next and then click save.

Note that if you want to continue to use the same configuration as you have tested the early beta with, you need to remove the “oauth-url” option from the “Advanced options” to make it use the default production oauth handler web service, instead of the beta service that was set up for previous testing of this change. Also remember that you need to generate a completely new CLI token and AuthID, as described above, and replace the existing value in the “AuthId” option.

ning42 · June 13, 2022, 4:01am

great news!

Is there a way to suppress the “warning”

[Warning-Duplicati.Library.Main.Controller-UnsupportedOption]: The supplied option --auth-username is not supported and will be ignored

FreaQ · June 13, 2022, 6:09am

You need to edit the target url.
Edit your backup → step 2 (target) → click on the three dots → copy target url to clipboard → three dots → import target url → paste url → delete the unsupported options → save

jottacloud://folder/subfolder/?auth-username=xxx&auth-password=yyy

ning42 · June 13, 2022, 3:33pm

Perfect - thank you!

w.geronius · June 13, 2022, 6:46pm

there is an even simpler method:
edit backup → target destination: switch ‘storage type’ to ‘Local Folder or Drive’, remove contents of ‘Username’ and ‘Password’, switch back to storage type ‘Jottacloud’ and continue

Many thanks to all contributors !!! Your hard efforts are very much appreciated!
My tests so far went all fine - backed up and restored small to medium sized batches

shalmirane · June 13, 2022, 7:17pm

Works for me as well

NickLavtchiev · June 13, 2022, 8:32pm

Here’s to confirm - it works smooth so far. Thanks for all the time and efforts to all the contributors who made our day to use Duplicati with Jottacloud. Hats down, guys

Edit: What’s the expiration time of this token?

shalmirane · June 13, 2022, 11:28pm

Is there a way how to update tokens in all configs?

I’ve about 8-10 (still no sort/order function in duplicati ) individual backup configurations - different timing, different folders,… - that all use the same jottacloud connection.

Some of those cover a lot of data (TBs) and their token expires during the operation - e.g. after all the time without backups, deleting old ones took over an hour, invalidating the token.

Now I can easily generate a new CLI and new OAuth token from it, but I got to replace it manually in each of those backup configurations each time that happens, otherwise, they won’t run on their normal schedule.