504 errors the past two days backing up to OneDrive v2

Support
1

Starting yesterday, my backups to OneDrive have been failing with errors of the for

StatusCode: 504, ReasonPhrase: ‘Gateway Timeout’,

One such result, from a few minutes ago:

Duplicati.Library.Backend.MicrosoftGraph.MicrosoftGraphException: GatewayTimeout: Gateway Timeout error from request https://graph.microsoft.com/v1.0/me/drive/root:/Selected_P_Folders:/children
Method: GET, RequestUri: ‘https://graph.microsoft.com/v1.0/me/drive/root:/Selected_P_Folders:/children’, Version: 1.1, Content: , Headers:
{
User-Agent: Duplicati/2.0.4.5
Authorization: Bearer ABC…XYZ
}
StatusCode: 504, ReasonPhrase: ‘Gateway Timeout’, Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Transfer-Encoding: chunked
request-id: 534ecdb6-4259-4ce2-bf69-b840181f0841
client-request-id: 534ecdb6-4259-4ce2-bf69-b840181f0841
x-ms-ags-diagnostic: {“ServerInfo”:{“DataCenter”:“South Central US”,“Slice”:“SliceC”,“Ring”:“5”,“ScaleUnit”:“003”,“RoleInstance”:“AGSFE_IN_3”,“ADSiteName”:“SCU”}}
Duration: 20009.39
Strict-Transport-Security: max-age=31536000
Cache-Control: private
Date: Wed, 21 Aug 2019 19:33:26 GMT
Content-Type: application/json
}
{
“error”: {
“code”: “UnknownError”,
“message”: “”,
“innerError”: {
“request-id”: “534ecdb6-4259-4ce2-bf69-b840181f0841”,
“date”: “2019-08-21T19:33:26”
}
}
}
at Duplicati.Library.Main.BackendManager.List()
at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.b__0(BackupResults result)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
at Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

Another, from yesterday:

System.AggregateException: One or more errors occurred. —> System.AggregateException: Object reference not set to an instance of an object. —> System.NullReferenceException: Object reference not set to an instance of an object.
at Duplicati.Library.Main.Volumes.BlockVolumeWriter.AddBlock(String hash, Byte data, Int32 offset, Int32 size, CompressionHint hint)
at Duplicati.Library.Main.Operation.Backup.SpillCollectorProcess.<>c__DisplayClass0_0.<b__0>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CoCoL.AutomationExtensions.d__10`1.MoveNext()
— End of stack trace from previous location where exception was thrown —

(more stack trace after this,then)

—> (Inner Exception #1) System.AggregateException: One or more errors occurred. —> Duplicati.Library.Backend.MicrosoftGraph.MicrosoftGraphException: GatewayTimeout: Gateway Timeout error from request https://graph.microsoft.com/v1.0/me/drive/root:/Tim_Selected_K_Folders/duplicati-bbe8bb0d83c1344a4b6dcaece72cda6fd.dblock.zip.aes:/createUploadSession
Method: POST, RequestUri: ‘https://graph.microsoft.com/v1.0/me/drive/root:/Tim_Selected_K_Folders/duplicati-bbe8bb0d83c1344a4b6dcaece72cda6fd.dblock.zip.aes:/createUploadSession’, Version: 1.1, Content: , Headers:
{
User-Agent: Duplicati/2.0.4.5
Authorization: Bearer ABC…XYZ
}
StatusCode: 504, ReasonPhrase: ‘Gateway Timeout’, Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Transfer-Encoding: chunked
request-id: 8db412ed-0b48-4691-9a35-a62422b05956
client-request-id: 8db412ed-0b48-4691-9a35-a62422b05956
x-ms-ags-diagnostic: {“ServerInfo”:{“DataCenter”:“South Central US”,“Slice”:“SliceC”,“Ring”:“4”,“ScaleUnit”:“000”,“RoleInstance”:“AGSFE_IN_0”,“ADSiteName”:“SCU”}}
Duration: 10058.0562
Strict-Transport-Security: max-age=31536000
Cache-Control: private
Date: Tue, 20 Aug 2019 17:38:27 GMT
Content-Type: application/json
}
{
“error”: {
“code”: “UnknownError”,
“message”: “”,
“innerError”: {
“request-id”: “8db412ed-0b48-4691-9a35-a62422b05956”,
“date”: “2019-08-20T17:38:27”
}
}
}

Any wisdom? Backups to other backends (Google, AWS S3) work fine.

2

is about the only thing I recognize from the info. While listing has been known to fail at 5000 files, I think it breaks differently (if it still breaks at all). You could read “BackendStatistics” “KnownFileCount” in job log.

This is a server-side error, and possibly Microsoft will just fix it. FWIW it went fine on my test yesterday…

If yours continues, I guess you could try further debugging, with some of the following actions to isolate it:

  • Try Destination screen “Test connection” button which usually does a list. Does it work? Is it quick? OneDrive operations time out at 100 seconds unless –http-operation-timeout is raised, but a list is usually a pretty small amount of data compared to the many megabytes that an upload would take.

  • While looking at Destination screen, open another tab and start down direct restore path which will probably need to do a file list to even get as far as showing you the dropdown of versions. Could it?

  • Duplicati.CommandLine.BackendTool.exe can also get a file list. Maybe code differs a bit from GUI.

  • Try third-party tools such as Cyberduck or rclone. They need to set up their own OneDrive access.

  • You could see if getting Duplicati new access to OneDrive helps, but bug doesn’t “look” like access.

3

Sorry for the extended lack of followup.

As predicted, it magically fixed itself. Yay, I guess.

Thank you for the wisdom.