A WebException with status RequestCanceled was thrown

1

I’ve been using Duplicati without issue for a couple years now. Until recently it’s been running nightly backups, flawlessly, in a dedicated vm.

About 4 weeks ago, shortly after upgrading to 2.0.7.103_canary_2024-04-19 (don’t know if there is a correlation), Duplicati started failing once a week, with a rather generic request canceled error.

The day it first failed, manually re-running the backup would produce the same error. However, the next day, the error was gone and Duplicati was running fine again.

The next Saturday morning, boom same error, but the next day all was fine.

Today, Duplicati just failed two days in a row. OK, so now I need to take a closer look at the problem.

To troubleshoot, I’ve done the following, which all succeeds:

  • Repair
  • Verify
  • Compact

Back when the error first occurred, I had previously increased http-operation-timeout and http-readwrite-timeout to 1 hour each.

Here is what seems to be the relevant part of the error message. Full log can be viewed here.

2024-06-01 23:28:09 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bc9d53dd22d98468e8eb1d8166805912d.dblock.zip.aes (49.95 MB)
2024-06-01 23:29:09 -05 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-b9a7caf6319434f439a78652bdf5fcd1c.dblock.zip.aes attempt 11 of 11 failed with message: A WebException with status RequestCanceled was thrown.
Amazon.Runtime.AmazonServiceException: A WebException with status RequestCanceled was thrown. ---> System.Net.WebException: The request was aborted: The request was canceled.

I am using iDriveE2, which uses Amazon S3 backend. It seems the backend storage decides it isn’t going to accept the upload.

I just lowered asynchronous-concurrent-upload-limit down to 3. Time will tell if this helps.

I’d appreciate any other suggestions.

UPDATE:
Backup just failed with asynchronous-concurrent-upload-limit=3, and then again with asynchronous-concurrent-upload-limit=1.

2

I just realized that I still had the original vm lying around, where I set up my first Duplicati backup for testing purposes. I fired this vm up, verified the older version of Duplicati was still running, ran the backup, and it failed in the exact same way.

I am going to reach out to E2 support as I’m not sure there is anything I can do on my end to resolve this.

3

Don’t you hate it when you come across a thread that did not come to some kind of resolution?

I certainly do.

Even though the issue ended up not being a Duplicati issue, I thought I document the experience.

I am trying to keep the story short, but after much testing, the issue seemed to be associated with my internet ip. It must have gotten flagged somehow, but for what I don’t know.

I was able to run a Duplicati backup just fine from my laptop, when connected to my Verizon jetpack. Switch back to my home network, however, and the backup failed with the same WebException error.

It was only PUT commands that were failing. I could connect to the endpoint and read from it just fine… that seemed to rule out blockage from my ISP.

Oh, and the problem was isolated to S3 buckets created in the Chicago region. If I created a bucket in a different region, backups worked just fine.

Idrive support was not very helpful. I don’t think they know what is going on.

I am currently using rclone to move 2.88TB of my data out of the bucket in the Chicago region to a bucket I created in Virginia. This is a very slow process, however.

This does not leave me with a lot of confidence in my cloud storage provider.

1 Like