Google Drive 403 errors during file verification process


#1

I’m getting lots of 403 error from a backup job when it is verifying data.

The job is using Google Drive as the destination. It can upload files, but it cannot download any for verification.

Backup started at 05/11/2018 19:07:26
Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  35573 files need to be examined (6.85 GB)
  29853 files need to be examined (6.82 GB)
  24274 files need to be examined (6.66 GB)
  18715 files need to be examined (5.91 GB)
  13068 files need to be examined (3.89 GB)
  7186 files need to be examined (2.54 GB)
  773 files need to be examined (266.27 MB)
  0 files need to be examined (0 bytes)
  Uploading file (1.09 KB) ...
  Uploading file (925 bytes) ...
  Uploading file (2.66 MB) ...
No remote filesets were deleted
Compacting not required
Checking remote backup ...
  Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
  Downloading file (2.66 MB) ...
  Downloading file (1.40 MB) ...
  Downloading file (1.56 MB) ...
Operation Get with file duplicati-b1b4026cbbdec40a2b609b9c08407ae05.dblock.zip.aes attempt 1 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (1.56 MB) ...
Operation Get with file duplicati-b1b4026cbbdec40a2b609b9c08407ae05.dblock.zip.aes attempt 2 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (1.56 MB) ...
Operation Get with file duplicati-b1b4026cbbdec40a2b609b9c08407ae05.dblock.zip.aes attempt 3 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (1.56 MB) ...
Operation Get with file duplicati-b1b4026cbbdec40a2b609b9c08407ae05.dblock.zip.aes attempt 4 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (1.56 MB) ...
Operation Get with file duplicati-b1b4026cbbdec40a2b609b9c08407ae05.dblock.zip.aes attempt 5 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
Failed to process file duplicati-b1b4026cbbdec40a2b609b9c08407ae05.dblock.zip.aes => The remote server returned an error: (403) Forbidden.
  Duration of backup: 00:03:27
  Remote files: 468
  Remote size: 19.76 GB
  Total remote quota: 21.78 TB
  Available remote quota: 11.78 TB
  Files added: 0
  Files deleted: 0
  Files changed: 1
  Data uploaded: 2.66 MB
  Data downloaded: 4.07 MB
Backup completed successfully!

I tried a new AuthID as suggested here: The remote server returned an error: (403) Forbidden with google drive but it did not help.

I have other backup jobs configured to upload to the same Google Drive account that still work perfectly. They can verify files just fine.

I’m using 2.0.3.3_beta_2018-04-02.

Do I need to reupload like @kenkendk mentioned in the linked post? How would I do that from the Linux command line? mono Duplicati.CommandLine.BackendTool.exe said that Duplicati.CommandLine.BackendTool.exe does not exist.

A couple more technical details:

The job I’m having issues with is running via a Debian Stretch based docker container. The job that works just fine is on Ubuntu 14.04, I made sure it was updated with the same Duplicati version.

My local machine I’d be using to reupload, if that is the answer, is on Ubuntu 18.04.

Edit:

Ok, now it’s not just the one job having issues. I ran the verify command on a 2.0.2.15_experimental_2018-01-03 instance I got this:


  Listing remote folder ...
  Downloading file (1.60 MB) ...
  Downloading file (289.92 KB) ...
Operation Get with file duplicati-ib78f3e6f887045fdad3be7bfa23b05df.dindex.zip.aes attempt 1 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (289.92 KB) ...
Operation Get with file duplicati-ib78f3e6f887045fdad3be7bfa23b05df.dindex.zip.aes attempt 2 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (289.92 KB) ...
Operation Get with file duplicati-ib78f3e6f887045fdad3be7bfa23b05df.dindex.zip.aes attempt 3 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (289.92 KB) ...
Operation Get with file duplicati-ib78f3e6f887045fdad3be7bfa23b05df.dindex.zip.aes attempt 4 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (289.92 KB) ...
Operation Get with file duplicati-ib78f3e6f887045fdad3be7bfa23b05df.dindex.zip.aes attempt 5 of 5 failed with message: The remote server returned an error: (403) Forbidden. => The remote server returned an error: (403) Forbidden.
  Downloading file (90.81 MB) ...
Failed to process file duplicati-ib78f3e6f887045fdad3be7bfa23b05df.dindex.zip.aes => The remote server returned an error: (403) Forbidden.
Return code: 3

This is a different backup job, but the same Google Account.


#2

The most likely cause for this, is that Google somehow flags the files as not belonging to Duplicati.
If you use the Google Drive (limited) option when creating the AuthID, Duplicati will only be able to read files it has created itself. If you somehow touch the files, Google will mark them as no longer belonging to Duplicati, and AFAIK you cannot change that back. The only workaround I know of, is to re-upload the files with the limited auth-id, which will mark them correctly again.

The Duplicat.CommandLine.BackendTool.exe is usually found in /usr/lib/Duplicati on Linux.

You can use the Google Drive (full access) option when creating the AuthID, which will give Duplicati full access to all files in the account, altough I recommend not using that option as it violates the “least-privileges” precaution.


#3

Finally got around to trying to figure this out. And the 403 errors have gone away. I even did a full verification job overnight.

So I’m still mystified about what happened, but at least the problem is gone.

I just wanted to post this for anyone finding this topic.


#4

Thanks for letting us know it magically fixed itself! :slight_smile:

I went ahead and flagged your post as the “solution”, but feel free to let us know if you disagree - or if it happens again!


#5

403 Errors on GDrive can be a symptom of a temporary ban (you either uploaded more than ~750Gb/day or you hit a mystical API limit). These bans last 24 hours - simply waiting should fix it.


#6

You wouldn’t happen to know if there’s an API way to verify that’s the cause of the 403 error, would you?


#7

Apparently there is no way do determine programatically that you are banned (best case is that it’s a 404 - which is of course rather a wide catch).