Duplicati job crashes when transfer rates fluctuate

I was hoping that this is just a problem with amazon, but unfortunately, it seems to be an duplicati problem too. Whatever Amazon might be doing (rate limiting or so), Duplicati can’t handle it and basically crashes the backup job without any notice as reported here:

So here is what happens: I have a backup job from a network drive to Amazon Cloud Drive and the upload keeps failing. Yesterday, it stopped sometime in the morning. Today it was in the afternoon:

As you can see in screenshot, I restarted the job at 22:28 and it started uploading again. It’s moving xtremely slow and I’m pretty sure that is Amazon’s fault, so my point here is: shouldn’t duplicati be able to handle whatever amazon is doing?

I can add that I have a cloud to cloud transfer going on at the same time and to the same amazon account (using rclone) and that has been running through the entire day. More precisely: I cannot say whether it was also slowed down in the afternoon (I’d actually assume that it was), but if it was, it nevertheless survived because when I checked it in the evening, it was uploading at high rates.

So my interpretation of the situation is that rclone can do something that duplicati can’t. And in addition, duplicati doesn’t issue any warning or so, it just dies silently.

Edit: I noticed in the rclone logs that it sometimes says

 Failed to copy: failed to open source object: HTTP code 429: "429 Too Many Requests": response body: "{}"

I don’t know if this is the only thing that is going on, but it might be a good point to start. How does duplicati deal with 429 responses?

Sorry, I don’t have any insight into this other than to say the “429 Too Many Requests” error was also reported here:

1 Like

It just treats it as a transfer error. meaning that it pauses and retries later.

Yes it should. Not sure why it does not just time out at some point.