Onedrive forcibly closed whilst counting down already backed up files

Support
1

2.0.6.104
Windows 2012 server running as a service
OneDrive v2

We have a backup setup and have been struggling for a long time now to get a backup, the source is around 500GB, we are using 250mb volume size but keep getting a remove connection forcibly closed error.

Testing with smaller data has shown us that the initial backup of a small backup, 30GB will be perfect for the first run with a new DB and nothing in OneDrive.
We then add another 30GB to the backup set and it will ALWAYS fail.
the backup starts, verifies backend then starts to count backup files no issues.
We then get to the backup, it will count down files already backed up but as soon as it reaches NEW data we immediately see the first 250mb files fail and start 1/6 reties this will continue until the 6/6 and fail the whole backup with the remote connection was forcibly closed.

it almost like a onedrive timeout is reached and the connection closed before any new files need uploading, please can anyone suggest a solution as we have been looking for over a month now with no answers.

2

Depending on your upload speed, a 250 MB volume may need increased http-operation-timeout.
Default of 100 seconds can be too low for oversized volumes. This doesn’t explain all symptoms.
It’s just an easy thing to try. You can also look on OneDrive to see if you have 250 MB files there.

3

Out of curiosity, where is this default configured ? I have tried to read the code and come up with this:

usually it means infinity, as could show this:

well, it could boil down to system defaults and here it’s a morass of uncertainty…

4

HttpClient.Timeout Property (Microsoft) is where I landed after trying to follow what source was using.
I suspect a 100 second timeout is a OneDrive (technically Graph) exclusive. Below has some history:

Allow MS Graph backends (e.g., OneDrive v2) to use OAuthHelper instead of OAuthHttpClient #4145
talks about differences and reasons why. I don’t recall source code use well, but you can explore that.

I remember wondering (as you did) if other timeouts were very long (infinite), which could also be bad.
Possibly it’s even behind some of the hangs reported. We can’t control all of them, as some are in libs.

5

uh. Thanks for the precision, but is this:

not setting the timeout to infinite by default on Windows at least (oAuthHttpClient is not used on Linux) ?

6

In addition to not being a C# developer, I’m not hugely familiar with .NET Framework timeout behavior. Several reported cases have been solved by increasing the OneDrive timeout. It’s not highly common.
Rather than try to debug by inspection (without debuggers), let’s hear from @Richard_Monckton first.

EDIT 1:

If this one is not a timeout problem, then please look at About → Show log → Live → Retry or set up a
log-file=<path> log-file-log-level=retry (might wind up at verbose or higher later) to see what happened.

EDIT 2:

The most solid evidence that this timeout is not infinite (though I can’t diagram the code path) is seeing “HTTP timeout 00:01:40 exceeded.” as seen here, which was a download. Current one may be upload, however without more logging (or a timeout tweak) we don’t even know for sure that this was a timeout.

I think this OneDrive timeout at 1:40 (100 seconds) is on upload, and increasing timeout solved timeout.
I’m citing experimental test results here. Possibly the cited code isn’t run here, or doing what one thinks.

7

i have tried to remove all cloud files and deleted the local db, increased from 250 to 500MB the backup will not upload ANY file now, i am now getting this:

20 Sep 2022 10:24: Operation Put with file duplicati-b4a2fc919c48f4cc9afc203a5ca8ad9e0.dblock.zip.aes attempt 6 of 6 failed with message: An error occurred while sending the request.
{“ClassName”:“System.Net.Http.HttpRequestException”,“Message”:“An error occurred while sending the request.”,“Data”:null,“InnerException”:{“ClassName”:“System.Net.WebException”,“Message”:“The underlying connection was closed: An unexpected error occurred on a send.”,“Data”:null,“InnerException”:{“ClassName”:“System.IO.IOException”,“Message”:“Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.”,“Data”:null,“InnerException”:{“NativeErrorCode”:10054,“ClassName”:“System.Net.Sockets.SocketException”,“Message”:“An existing connection was forcibly closed by the remote host”,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:" at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)\r\n at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)“,“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:“8\nEndReceive\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Net.Sockets.Socket\nInt32 EndReceive(System.IAsyncResult)”,“HResult”:-2147467259,“Source”:“System”,“WatsonBuckets”:null},“HelpURL”:null,“StackTraceString”:” at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)\r\n at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)“,“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:“8\nEndWrite\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Net.TlsStream\nVoid EndWrite(System.IAsyncResult)”,“HResult”:-2146232800,“Source”:“System”,“WatsonBuckets”:null},“HelpURL”:null,“StackTraceString”:” at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)“,“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:“8\nEndGetResponse\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Net.HttpWebRequest\nSystem.Net.WebResponse EndGetResponse(System.IAsyncResult)”,“HResult”:-2146233079,“Source”:“System”,“WatsonBuckets”:null},“HelpURL”:null,“StackTraceString”:” at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Duplicati.Library.OAuthHttpClient.d__6.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Duplicati.Library.Backend.MicrosoftGraphBackend.d__76.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at Duplicati.Library.Backend.MicrosoftGraphBackend.d__60.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Duplicati.Library.Main.Operation.Backup.BackendUploader.d__24.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Duplicati.Library.Main.Operation.Backup.BackendUploader.<>c__DisplayClass20_0.<b__0>d.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at Duplicati.Library.Main.Operation.Backup.BackendUploader.d__21.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at Duplicati.Library.Main.Operation.Backup.BackendUploader.d__21.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Duplicati.Library.Main.Operation.Backup.BackendUploader.d__20.MoveNext()\r\n— End of stack trace from previous location where exception was thrown —\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Duplicati.Library.Main.Operation.Backup.BackendUploader.d__18.MoveNext()",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:“8\nThrow\nmscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Runtime.ExceptionServices.ExceptionDispatchInfo\nVoid Throw()”,“HResult”:-2146233088,“Source”:“mscorlib”,“WatsonBuckets”:null,“SafeSerializationManager”:{“m_serializedStates”:[{}]},“CLR_SafeSerializationManager_RealType”:“System.Net.Http.HttpRequestException, System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”}

8

This seems like the test is going in the wrong direction. What about smaller remote volume size,
or larger http-operation-timeout? I used to run one with the following value, due to slow uploads:

image

I wish the stack trace was a little more informative, but let’s make sure there’s time to do uploads.

If you don’t want to change the timeout, what can upload at default remote volume size of 50 MB?

9

I found below while Google searching for similar stack traces (not having much luck yet). Is it related?

10

ok, reduced to 50MB and still will not even upload the first 50mb, checked the http-operation-timeout and i have it set to 100minutes to try and get over this. this is just so odd.

11

Are these the servers that went through a PCI lockdown, which you thought you had fixed (maybe not)?

So now that it’s known that small files can fail too, next step might be to try driving some precise testing.

Export As Command-line will get a URL which you can edit to a blank folder to either do automated test

Duplicati.CommandLine.BackendTester.exe

or manual test to see whether you can find some operations that work, and some that give you the error

Duplicati.CommandLine.BackendTool.exe

12

In addition to the above-mentioned questions and tests, you can try some Duplicati GUI tests.
Test connection button on Destination screen does a list operation but just wants to see it run.
Verify files button on home page does a list, checks it (can complain) then samples download.
You can try changing allowed-ssl-versions to see if it helps with those. PCI may want TLS 1.2.

Microsoft has been busy killing off Internet Explorer, but if you have it, can it access OneDrive?
There are some settings in there somewhere if you want to see which TLS versions that needs.

What about OneDrive as viewed in File Explorer?

Each of these methods might use a different connection point, but maybe some will give a clue.

13

in the GUI test connection works ok, verify works ok i will test some command line items later, seems i cannot write any at the moment, hopefully the command line with help me.

14

so command line testing, i can LIST every time but as soon as i try a PUT i get the error the underlying connection was closed, unable to read data from the transport connection

15

running the backed tester it is the same story, cannot PUT a file:

Uploading file 0, 31.39 MB … Failed to upload file 0, error message: System.AggregateException: One or more errors occ
urred. —> System.Net.Http.HttpRequestException: An error occurred while sending the request. —> System.Net.WebExcept
ion: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Unable to
read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net
.Sockets.SocketException: An existing connection was forcibly closed by the remote host.

i then get a lot of errors saying files cannot be deleted because they cannot be found, i guess that is due to them not being uploaded.

16

right, the plot thickens, install th eonedrive client and setup with the same account being used for the backup, cannot upload a file, just sits there thinking and thinking, i will remove the account and setup a fresh one as potential that there is an issue with the onedrive account

17

ok, might have something, new account made no difference, i cannot log in with IE, i get:

Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to https://www.office.com again. If this error persists, it is possible that this site uses an unsupported protocol or cipher suite such as RC4 (link for the details), which is not considered secure. Please contact your site administrator.

i have check TLS 1.0 1.1 & 1.2 are enabled in advanced settings

18

Hello

could this be relevant ?

19

About TLS and ciphers, is this a system that underwent PCI lockdown where you suspected that?
Windows Ciphers causing backup to fail to onedrive. There are directions for trying to select TLS.
PCI has also been known to disable weak ciphers, and I’d suspect that an older system has some.

I posted an example of using Wireshark to get a nice view of the cipher situation, but I will say that
the problem with this theory is that a certificate failure seems like it should stop list, not just upload.
The Internet Explorer failure is more thorough. Duplicati’s partially working status is a little stranger.

The way this topic began, there were some, so upload was working then. What may have changed?
Do you recall any file sizes or dates? For upload testing, I would suggest starting with very small file.
Upload or replace the contents of a DriveItem says beyond 4 MB is done differently. I haven’t tested.

Can BackendTool download (get) a file if you manage to point to an existing one or get one in there?
What sort of OneDrive is this? That’s kind of a brand name, with different packaging and technology.
Is it paid up and with enough free space? Not having that might get you in an uploads-blocked status.

How locked down is this system and its environment? Can you install other browsers for some tests?
Corporate networks, antivirus programs, and data exfiltration prevention might interfere with requests.
You can get a clue if something is in the middle by looking at a web site’s certificate to read its issuer.

20

Well, the OP said in the first post it was v2 and AFAIK there are only 2 brands, v2 and Pro. Free OneDrive V2 size is 5 Gb - you need only to provide a mail address to use it, I got one to test Duplicati with OAuth. Obviously it can’t be the free version here.