Ssh disconnection leading to Unexpected difference in fileset?

Support
1

Hello,

I run duplicati on 2 systems, one Ubuntu 18.4 LTS and one Window 10; some backups to en external disk, some to a remote fileserver via SSH. It seems all(at least most) my occurences of the dreaded “Unexpected difference in fileset” errors occur with this SSH connection related error. This is what I find in the logs:
System.AggregateException: One or more errors occurred. ---> System.AggregateException: The channel "BackendRequests" is retired ---> CoCoL.RetiredException: The channel "BackendRequests" is retired

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at CoCoL.Channel`1.<WriteAsync>d__32.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()

   --- End of inner exception stack trace ---

   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()

   --- End of inner exception stack trace ---

   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)

   at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result)

   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)

   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

---> (Inner Exception #0) System.AggregateException: The channel "BackendRequests" is retired ---> CoCoL.RetiredException: The channel "BackendRequests" is retired

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at CoCoL.Channel`1.<WriteAsync>d__32.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()

   --- End of inner exception stack trace ---

   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()

---> (Inner Exception #0) CoCoL.RetiredException: The channel "BackendRequests" is retired

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at CoCoL.Channel`1.<WriteAsync>d__32.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()<---



---> (Inner Exception #1) System.AggregateException: One or more errors occurred. ---> Renci.SshNet.Common.SshConnectionException: Client not connected.

   at Renci.SshNet.Session.SendPacket(Byte[] packet, Int32 offset, Int32 length)

   at Renci.SshNet.Session.SendMessage(Message message)

   at Renci.SshNet.Channels.Channel.SendData(Byte[] data, Int32 offset, Int32 size)

   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)

   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)

   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)

   at Duplicati.Library.Backend.SSHv2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<DoPut>d__21.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<>c__DisplayClass17_0.<<UploadFileAsync>b__0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<DoWithRetry>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<DoWithRetry>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<UploadFileAsync>d__17.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<UploadBlockAndIndexAsync>d__15.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()

   --- End of inner exception stack trace ---

---> (Inner Exception #0) Renci.SshNet.Common.SshConnectionException: Client not connected.

   at Renci.SshNet.Session.SendPacket(Byte[] packet, Int32 offset, Int32 length)

   at Renci.SshNet.Session.SendMessage(Message message)

   at Renci.SshNet.Channels.Channel.SendData(Byte[] data, Int32 offset, Int32 size)

   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)

   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)

   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)

   at Duplicati.Library.Backend.SSHv2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<DoPut>d__21.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<>c__DisplayClass17_0.<<UploadFileAsync>b__0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<DoWithRetry>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<DoWithRetry>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<UploadFileAsync>d__17.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<UploadBlockAndIndexAsync>d__15.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<<Run>b__13_0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at CoCoL.AutomationExtensions.<RunTask>d__10`1.MoveNext()<---

<---

<---

I have 2 questions about this:

  1. How is a disconnect to the backend handled? First of all, the way this failure is reported seems very much like an unhandled error. The backup or even delete and repair db operations seems to fail with just “One or more errors occured” message (popup in UI), without any helpful information. When going to the backup and looking at the logs about a specific backup, this type of errors isn’t mentioned there, it’s only mentioned in the general logs. Also, it does sound a bit odd to me that a file upload (or download) should fail in just 1 attempt. I consider it “normal” that connections get interrupted once in a while, especially large file transfers, that’s why we use tools that have resume support. So it’s a bit odd the entire operations fails just due to 1 disconnect.
  2. I now have a hunch that the “Unexpected difference in fileset” error occurs each time a backup operation did not complete due to this error. Does that seem plausible to anyone more familiar with the internals of Duplicati2?
2

–number-of-retries default should prevent that. About → Show log → Live → Retry can let you watch it.

What Duplicati version? Even latest Duplicati messes up errors such as retry exhaustion in compact, but the error is not the one you saw. It commonly is files that are missing from the remote storage.

Such destination operation problems have been handled well during the main backup for awhile, I believe.

Earlier Duplicati versions could get Unexpected difference in fileset from source files changing in middle of being backed up. The file growing further after a short read was seen caused confusion of EOF.

3

Hi,

Running Duplicati - 2.0.5.1_beta_2020-01-18, afaik the latest version. One of the backup profiles I created only a few weeks ago (basically an export+import of the existing backup, with primarily the storage location changed), so this particular backup has never ran with an older version of Duplicati2.
I doubt the files are changing during the backup, as the option “snapshot-policy” is set to “Required”.
In the “Remote log” I’m not seeing any re-occurences of filenames, so at first glance it doesn’t look like Duplicati is retrying. OTOH, the “General” log isn’t showing the failed attempts at all, so maybe the failed file transfers are not logged there either. I have not configured a value for number-of-retries, so the default 5 should be applied? Anyway, now rerunning a backup with the option set explicitly to 5.

4

Update: looking at the Live/Retry logs, I saw several messages of retries. Is it possible the 5 retry attempts are cumulative for all files and not per file? Seeing an attempt 6/5 fail I tried to find which upload failed 5 times, and matching with the access logs on the file server, I could only conclude that several archives needed a second attempt, but it didn’t look like any individual file was interrupted 5 times. Anyways, I upped the number-of-retries to 10 and the retry-delay to 30 seconds; and now it seems to work, both backup profiles succeeded.
I turned my investigation to the NAS server side as well, but I cannot find any indication on the server logs that it is disconnecting clients due to some error. So I’m not sure at all why Duplicati detects >5 times a disconnect from server. I have other SSH connections running over the same VPN connection, and they stay connected just fine. Even the Duplicati on my Ubuntu connects to the same NAS and runs (much larger) backups mostly fine.

5

You can’t go by that because the same file content gets a new name for each try. This agrees with any remotes that don’t allow possible file overwrites, and also lets uploads that got an error get deleted later.
Clicking on the line gets a file hash and size. Seeing same values implies retry. Here I got two at 5 tries.
This is an artificial test on an FTP server that I had handy, and which lets me turn it off and on as I like…

image
image1341×600 83.6 KB

No, and further evidence is in screenshot above. That’s an awkward way to see retries. A live log shows

image
image1327×764 132 KB

where you can see the failure, rename, and retry more clearly. I clicked on one failure to get some details.

The file rename might have thrown you. I did make mine artificially neater by using advanced option below:

image
image1309×143 9.47 KB

asynchronous-concurrent-upload-limit at default 4 would have all sorts of parallel uploads failing, which is much easier to follow by eye, compared to following 4 flailing uploads at the same time, all with renaming.

Could you please post an actual error from the live log, similar to example above, to show what it shows?
It would also be useful to find what fraction of uploads fail, and whether there is any visible pattern to fails.

My statistics are below (though I stopped the backup early). If yours ever finishes, look at Complete log:

image

Are you running at default 50 MB Remote volume size on screen 5? That option confuses some people.

Duplicati.CommandLine.BackendTester.exe can be fed a URL to an empty folder with URL modified from Export As Command-line value. Move it from the actual backup folder to an empty one for backend tester.

6

I’ve ran another backup manually, but one earlier must have ruined it again cause I got the unexpected difference in fileset.
Now I know the different filenames are actually retries of the same file, the results as shown here indicate some problematic issue with the ssh connection.

screenshot-localhost_8200-2020.06.30-00_48_00
screenshot-localhost_8200-2020.06.30-00_48_001841×7678 510 KB

I’ve started a test with the backend test tool, it runs into the same issues it seems:

Starting run no 0
Generating file 0 (7.56 MB)
Generating file 1 (9.08 MB)
Generating file 2 (43.61 MB)
Generating file 3 (41.45 MB)
Generating file 4 (29.89 MB)
Generating file 5 (39.02 MB)
Generating file 6 (10.45 MB)
Generating file 7 (29.61 MB)
Generating file 8 (31.37 MB)
Generating file 9 (43.13 MB)
Uploading wrong files ...
Generating file 10 (1.55 KB)
Uploading file 0, 1.55 KB ...  done!
Uploading file 0, 1.55 KB ...  done!
Uploading file 9, 1.55 KB ...  done!
Uploading files ...
Uploading file 0, 7.56 MB ...  done!
Uploading file 1, 9.08 MB ... Failed to upload file 1, error message: Renci.SshNet.Common.SshConnectionException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
   --- End of inner exception stack trace ---
   at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
   at Renci.SshNet.Channels.Channel.GetDataLengthThatCanBeSentInMessage(Int32 messageLength)
   at Renci.SshNet.Channels.Channel.SendData(Byte[] data, Int32 offset, Int32 size)
   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)
   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)
   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)
   at Duplicati.Library.Backend.SSHv2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)
   at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming), remote name: UEoSdLcO7hINPW5p3lWQNqNRfL5JXV9j7cER1yiV3aBe4b1guBiISqiDZGNAQBZ
  Inner exception: System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
Uploading file 2, 43.61 MB ... Failed to upload file 2, error message: Renci.SshNet.Common.SshConnectionException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
   --- End of inner exception stack trace ---
   at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
   at Renci.SshNet.Channels.Channel.GetDataLengthThatCanBeSentInMessage(Int32 messageLength)
   at Renci.SshNet.Channels.Channel.SendData(Byte[] data, Int32 offset, Int32 size)
   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)
   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)
   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)
   at Duplicati.Library.Backend.SSHv2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)
   at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming), remote name: 2BTtofcjuxnnrkujOxVLw49ZpPKNFzTu
  Inner exception: System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
Uploading file 3, 41.45 MB ...  done!
Uploading file 4, 29.89 MB ...  done!
Uploading file 5, 39.02 MB ... Failed to upload file 5, error message: Renci.SshNet.Common.SshConnectionException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
   --- End of inner exception stack trace ---
   at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
   at Renci.SshNet.Channels.Channel.GetDataLengthThatCanBeSentInMessage(Int32 messageLength)
   at Renci.SshNet.Channels.Channel.SendData(Byte[] data, Int32 offset, Int32 size)
   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)
   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)
   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)
   at Duplicati.Library.Backend.SSHv2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)
   at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming), remote name: W4KgDDDj1qv81RRd1OYIzgGxOAim3uYVSLdjjUwdHq5jidKo
  Inner exception: System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
Uploading file 6, 10.45 MB ... Failed to upload file 6, error message: Renci.SshNet.Common.SshConnectionException: Client not connected.
   at Renci.SshNet.Session.SendPacket(Byte[] packet, Int32 offset, Int32 length)
   at Renci.SshNet.Session.SendMessage(Message message)
   at Renci.SshNet.Channels.Channel.SendData(Byte[] data, Int32 offset, Int32 size)
   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)
   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)
   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)
   at Duplicati.Library.Backend.SSHv2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)
   at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming), remote name: ewvbO3l9IIBZugTiskQETxNC6oRWSN
Uploading file 7, 29.61 MB ...  done!
Uploading file 8, 31.37 MB ... Failed to upload file 8, error message:     Renci.SshNet.Common.SshConnectionException: An existing connection was forcibly closed by the remote host ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
   --- End of inner exception stack trace ---
   at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
   at Renci.SshNet.Channels.Channel.GetDataLengthThatCanBeSentInMessage(Int32 messageLength)
   at Renci.SshNet.Channels.Channel.SendData(Byte[] data, Int32 offset, Int32 size)
   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)
   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)
   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)
   at Duplicati.Library.Backend.SSHv2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)
   at Duplicati.CommandLine.BackendTester.Program.Uploadfile(String localfilename, Int32 i, String remotefilename, IBackend backend, Boolean disableStreaming), remote name: 9vyQqUoWCY29nA2rZhJZHyK8Jp5vV5VQLpeNxbd
  Inner exception: System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
   at Renci.SshNet.Abstractions.SocketAbstraction.Read(Socket socket, Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Renci.SshNet.Session.TrySocketRead(Byte[] buffer, Int32 offset, Int32 length)
   at Renci.SshNet.Session.ReceiveMessage()
   at Renci.SshNet.Session.MessageListener()
Uploading file 9, 43.13 MB ...  done!
Renaming file 1 from UEoSdLcO7hINPW5p3lWQNqNRfL5JXV9j7cER1yiV3aBe4b1guBiISqiDZGNAQBZ to oSZdaCOnsMzzgfQgwXQKVHolDtUvHTp3rrxVlx
Verifying file list ...
*** File with name 2BTtofcjuxnnrkujOxVLw49ZpPKNFzTu has size 45723479 but the size was reported as 13619424
*** File with name W4KgDDDj1qv81RRd1OYIzgGxOAim3uYVSLdjjUwdHq5jidKo has size 40911221 but the size was reported as 18071928
*** File with name ewvbO3l9IIBZugTiskQETxNC6oRWSN has size 10957388 but the size was reported as 8315706
*** File with name 9vyQqUoWCY29nA2rZhJZHyK8Jp5vV5VQLpeNxbd has size 32898081 but the size was reported as 28613886
*** File with name oSZdaCOnsMzzgfQgwXQKVHolDtUvHTp3rrxVlx has size 9525854 but the size was reported as 3077466
Downloading files
Downloading file 0 ... done
Checking hash ... done
Downloading file 1 ... done
Checking hash ... failed
*** Downloaded file was corrupt
Downloading file 2 ... done
Checking hash ... failed
*** Downloaded file was corrupt
Downloading file 3 ... done
Checking hash ... done
Downloading file 4 ... done
Checking hash ... done
Downloading file 5 ... done
Checking hash ... failed
*** Downloaded file was corrupt
Downloading file 6 ... done
Checking hash ... failed
*** Downloaded file was corrupt
Downloading file 7 ... done
Checking hash ... done
Downloading file 8 ... done
Checking hash ... failed
*** Downloaded file was corrupt
Downloading file 9 ... done
Checking hash ... done
Deleting files...
Checking DNS names used by this backend...

Now my first reaction as an IT-er would be: there’s something wrong with this server/your connection. But I’m running from this same physical machine, at the same time this backup tester is failing, a backup from Duplicati on Linux/Mono, to exactly the same remote server (so the failing is running from within a Windows VM). So my first question now is: is the SSH implementation in both the same? Or is a different implementaion used on windows and a native scp/sftp on Linux?

7

They’re both on SSH.NET which is portable C# code, however below that there’s mono on Linux and .NET Framework on Windows, and below that there are other differences. How to isolate? Can you do packets?