Cannot connect via SSH (Failed to connect: Session operation has timed out)

[DL38]

Hello everyone, I have started using Duplicati after CrashPlan’s end of life notice and have run into a problem connecting to a server via SSH.

I have Computer A (Windows 7, 64 bit) and Computer B (Windows 7, 32 bit) that I would like to back up to each other. Both have OpenSSH installed (Install Win32 OpenSSH · PowerShell/Win32-OpenSSH Wiki · GitHub) and I have been able to successfully connect to A (from B) and B (from A) via WinSCP. In WinSCP, I’m able to read/write files with no issues.

Using Duplicati, Computer A can connect to Computer B via SSH and backup fine. Unfortunately, Computer B cannot establish a connection to Computer A – the error message given at configuration is “Failed to connect: Session operation has timed out.”

I have checked my settings in Duplicati and Windows Firewall and everything appears to be the same on both Computers with the exception of the destination/credentials for each server. I can’t figure out what I’m doing wrong, so your feedback/suggestions are greatly appreciated. Thank you!

[JonMikelV]

Hello @DL38, welcome to the forum!

While it is unfortunate that Duplicati doesn’t have a built in point-to-point backup feature like CrashPlan did, it sounds like you’re going about putting together your own solution correctly.

Are computers A and B on the same network or is this going over the internet?

Depending on your Duplicati version you could try adding --verbose=true to your job and try running it from the command line. Or using --log-file=<path> and --log-file-log-level=profiling to get a file full of detail…

Oh, and do your firewalls and ssh servers have their own logs you could check just to see if they register an incoming connection?

I realize OpenSSH is not the same as WinSSHD, but when you set up your OpenSSH servers, did you use any of the info from this How-To topic?

How to Set Up WinSSHD SFTP Server On Windows

Summary I wanted to post a quick walkthrough on how to set up Bitvise WinSSHD on a Windows computer to use as a SFTP target/destination for Duplicati. This is a great enterprise class piece of software that the author is kind enough to allow us to use for personal use free of charge and is great when paired with Duplicati. Don’t let the length of this post scare you, the software is very easy to set up and run, but I wanted to be thorough for those that like step by step instructions. Basic I…

[DL38]

Hello @JonMikelV thank you for the response and apologies for the delayed reply!

@JonMikelV

Are computers A and B on the same network or is this going over the internet?

Yes computers A and B are on the same network at the moment. However the goal will be to do this across the internet as well.

@JonMikelV

Depending on your Duplicati version you could try adding --verbose=true to your job and try running it from the command line. Or using --log-file=<path> and --log-file-log-level=profiling to get a file full of detail…

I ran my job with logging and had the following message:

Operation List with file attempt 1 of 5 failed with message: Session operation has timed out
Renci.SshNet.Common.SshOperationTimeoutException: Session operation has timed out
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle)
at Renci.SshNet.Session.Connect()
at Renci.SshNet.BaseClient.Connect()
at Duplicati.Library.Backend.SSHv2.CreateConnection()
at Duplicati.Library.Backend.SSHv2.d__39.MoveNext()
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Duplicati.Library.Main.BackendManager.DoList(FileEntryItem item)
at Duplicati.Library.Main.BackendManager.ThreadRun()

When I was configuring the backup and tried to test the connection, the log yielded this:

Renci.SshNet.Common.SshOperationTimeoutException: Session operation has timed out
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle)
at Renci.SshNet.Session.Connect()
at Renci.SshNet.BaseClient.Connect()
at Duplicati.Library.Backend.SSHv2.CreateConnection()
at Duplicati.Library.Backend.SSHv2.d__39.MoveNext()
at Duplicati.Library.Interface.BackendExtensions.TestList(IBackend backend)
at Duplicati.Library.Backend.SSHv2.Test()
at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection(String url, RequestInfo info)
at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST(String key, RequestInfo info)
at Duplicati.Server.WebServer.RESTHandler.DoProcess(RequestInfo info, String method, String module, String key)

@JonMikelV

Oh, and do your firewalls and ssh servers have their own logs you could check just to see if they register an incoming connection?

Unfortunately my knowledge of how to do this is limited so going off of this site and enabling logging, I do not see an incoming connection from computer B to Computer A. However I can see an inbound connection from B to A when SSHing via WinSCP.

@JonMikelV

I realize OpenSSH is not the same as WinSSHD, but when you set up your OpenSSH servers, did you use any of the info from this How-To topic?
How to Set Up WinSSHD SFTP Server On Windows

I did reference this but I am not sure what else to do. I have successfully set up an SSH server on Computer B and have been backing up from A to B with no issues. I believe I set up the SSH server properly on A as I can connect/read/write via other means (putty/WinSCP) but for whatever reason, I can’t connect via Duplicati.

Thank you again for the assistance. I appreciate your help as I migrate away from CrashPlan and onto Duplicati!

[ts678]

Although Windows Firewall Logging (any other firewalls?) shows no incoming connection from B, you can try:

C:\>netstat -ano -p tcp | find ":22"
  TCP    0.0.0.0:22             0.0.0.0:0              LISTENING       3516
  TCP    192.168.0.110:59980    192.168.0.99:22        SYN_SENT        22464

C:\>

on B (maybe try it on A first to get the feel) to see if B is trying to make a connection. The above SYN_SENT will just time out because I intentionally configured an IP address with nothing listening. The LISTENING seen would be the SSH server on this machine. In a cross-connected situation, A and B should have similar views.
Depending on whether or not Duplicati is configured as a service, netstat might need Administrator privilege.

From the logs you supplied, the Connect() is clearly failing, but I’m not sure at what level, for example if there might be some problem with certificates. For diagnosis, some people find accept-any-ssl-certificate helps out.

[DL38]

Thank you, I tested the connection from B to A in Duplicati GUI and ran netstat and here’s what I found.

netstat -ano -p tcp | find “:22”
TCP 0.0.0.0:22 0.0.0.0:0 LISTENING 2620
TCP 192.168.1.14:8924 192.168.1.17:22 ESTABLISHED 1064

192.168.1.14 is Computer B and .17 is Comp A. PID 1064 corresponds to Duplicati.GUI.TrayIcon.exe

I added “accept-any-ssl-certificate” to my .bat and it’s timing out:

Backup started at 7/30/2018 9:14:37 PM
Checking remote backup …
Listing remote folder …
Operation List with file attempt 1 of 5 failed with message: Session operation
has timed out => Session operation has timed out

Thanks for the continued help.

[JonMikelV]

@DL38

I added “accept-any-ssl-certificate” to my .bat and it’s timing out:

Sorry if this was already covered, but evern with --accept-any-ssl-certificate it’s timing out the same way it was before, correct?

The fact that you can connect with Putty definitely points to a Duplicati/configuration issue, but since the GUI test also resulted in a timeout I’m guessing it’s not a certificate issue (they usually result in different errors).

You mentioned not seeing any incoming connection when using the logging - do you see the same thing when connecting successfully with Putty?

If you ALSO don’t see a connection when you know you are connected via Putty, then the connection log isn’t working the way want so can’t be relied on. But if it DOES show a connection with Putty (but not Duplicati) then that implies there’s something blocking Duplicati from even getting out of the machine it’s running on.

[ts678]

The netstat test was an attempt to see if Duplicati was getting out of B. The ESTABLISHED result says it is, at least at the TCP level. If the firewall log on A is still seeing differences between that and the other connections from other programs, that seems puzzling. Perhaps it could be checked again. Also, when using winscp, tests ideally would use sftp and not some other winscp protocol (such as scp) just in case that makes a difference.

Troubleshooting Steps including Logging as mentioned might also help see what (if anything) server on A got.

I’m sorry this is being difficult to track down. Thanks for trying various clients. There ARE also other servers…

On the off chance that it’s relevant, when I set up a test to help with a forum complaint about another provider, when I ran it I saw it trying to do my connection to the .99 address from this topic. Possibly I just ran the wrong job, but if there was some leftover from the failed previous one that got continued in a different job, that’s bad.

[ts678]

SFTP (SSH) has some options that may be useful. The most interesting one may be --ssh-operation-timeout

In the GUI, if I run"Test connection", I get a response in a fraction of a second. Does yours have much delay?

Key exchange is slow when size of group is more than 1024 bit (issue #304 and #130).

is a fix in 2016.1.0-beta4 of that. Duplicati’s last library update got it up to 2016.1.0-beta3, so it lacks that fix.

But before we go trying to manually update a library, it’d be good to know if you spot any slowness doing the “Test connection” or perhaps by watching the debug output if you try the Troubleshooting Steps I mentioned:

This will dump debug logs in real time to stdout on the console

[DL38]

@JonMikelV and @ts678 thank you for your helpful suggestions. I have tried your steps again and here’s the response to your latest posts:

@JonMikelV

Sorry if this was already covered, but evern with --accept-any-ssl-certificate it’s timing out the same way it was before, correct?

Yes, even with the --accept-any-ssl-certificate it is timing out the same as before, from what I can tell. I am clearly reaching the limit of my current understanding so please let me know if you spot something amiss. Here is the output after I run my .bat file with the backup config that I exported from Duplicati GUI

Input options:
backup-name: Testbackup7-30
dbpath: C:\Users\CompB\AppData\Local\Duplicati\ABCDEFG.sqlite
encryption-module: aes
compression-module: zip
dblock-size: 50mb
passphrase: test
disable-module: console-password-input
accept-any-ssl-certificate: true
verbose: true
ssh-operation-timeout: 0
Backup started at 8/4/2018 9:06:43 PM
Checking remote backup …
Listing remote folder …
Operation List with file attempt 1 of 5 failed with message: Session operation
has timed out => Session operation has timed out

@ts678 the ssh-operation-timeout did not yield any positve results here either.

@JonMikelV

You mentioned not seeing any incoming connection when using the logging - do you see the same thing when connecting successfully with Putty?

I check the Windows 7 Firewall again and actually do see an inbound connection when I test from B to A via the Duplicati config GUI.

Windows firewall logs:

With Putty: ALLOW TCP 192.168.1.14 192.168.1.17 10821 22 0 - 0 0 0 - - - RECEIVE
With Duplicati Test: ALLOW TCP 192.168.1.14 192.168.1.17 1076 22 0 - 0 0 0 - - - RECEIVE

Per @ts678’s suggestion I also checked the OpenSSH server log on A (in windows Event Viewer). There is nothing in the debug but curiously, in the operational log section, when connecting with putty:

sshd: Accepted publickey for ComputerB from 192.168.1.14 port 10821 ssh2: RSA SHA256:XXXXX

when connecting with Duplicati Test:

sshd: WARNING: could not open PROGRAMDATA\ssh/moduli (No such file or directory), using fixed modulus

Could this be part of the issue?

I also just tried connecting from a third computer (“C”) and C cannot connect to either A or B via Duplicati. It can connect via Putty. The OpenSSH server logs on A are the same (“accepted publickey for ComputerC…” for Putty and “WARNING: could not open…” for Duplicati.

Thanks for any additional suggestions!

[DL38]

@ts678

In the GUI, if I run"Test connection", I get a response in a fraction of a second. Does yours have much delay?

Yes there is slowness. When setting up a new connection from A to B, which is successful, it takes about 5 second to show the “Trust Host Certificate” popup, which reads

No certificate was specified previously, please verify with the server administrator that the key is correct: ssh-rsa 2048 XXXX Do you want to approve the reported host key?

For B to A, it stays on “Testing Connection” for about 20 seconds and then fails.

[ts678]

TL;DR You likely hit a known SSH.NET bug. Missing moduli file might have made it worse. Want to try .dll fix?

The asymmetry still bothers me, however I don’t know if computer speeds vary, e.g. perhaps B (on 32 bit OS) is slower than A. Nevertheless I’ll walk through the known speedup fix because it appears to match your use.

First look at SSH.NET issue Key exchange is slow when size of group is more than 1024 bits #304

Software is same as yours. “WinSCP SSH server” reference links to winscp.net then the site you mentioned.

Exception is same as yours. Both look like SshOperationTimeoutException from Session.cs

Next look at SSH.NET Session timeout caused by long time calculation on KeyExchangeDiffieHellman->PopulateClientExchangeValue #130

The enormous decimal number, when pasted into Linux dc to output as hex, looks a lot like the 8192-bit MODP Group which is also fixed into sshd to use when the moduli file is missing. Smaller values are also hardcoded, and the choice depends on the max requested. #304 shows 1024/min 8192/max requested.
Source of sshd shows this gets you the 8192 bit version because it only looks at max if moduli is missing.

Selection when moduli is there is complex. The man page says “sshd(8) then randomly selects a modulus from /etc/ssh/moduli that best meets the size requirement” so it’s not clear if it would have been any faster.

Next consider the missing moduli file, and what to do.

My Linux distro seems to have included an /etc/ssh/moduli file, but possibly your sshd did not bundle one. Possibly yours got lost somehow. You could compare your two systems, or check with your sshd provider.
Generating moduli is done with ssh-keygen, but see ssh-keygen -G modulus candidate generation failed

Bottom line here is that if you can figure out how to fix moduli easily, try it, otherwise maybe try a fixed .dll.
This is basically just backing up the one Duplicati ships, and replacing it with one downloaded in a zip file.
The challenges include making sure you replace the right one. Could you say what you installed (version
visible in Control Panel or Settings) and what version you’ve updated to (visible in Duplicati GUI in About)?

[DL38]

Hello @ts678 just wanted to say I saw your tag on the other post and don’t have any updates at the moment. Unfortunately I’ve been out of the country so haven’t have a chance to work on this. In any case I will be setting up a similar backup configuration for my parents and will note if there are any OpenSSH issues.

Thank you!

[Mugros]

I also get the time out error whenever I try to use a non-standard port.
I connect to my Synology NAS on port 22 and backup works. Synology advises to change the SSH port for security. I gave it a trying to connect to a non-standard port always times out.
Putty works, so it is likely a Duplicati issue.

[ts678]

Hello @Mugros and welcome to the forum!

@Mugros

I gave it a trying to connect to a non-standard port always times out.

It’s working fine here, changing port to 220 on the SFTP server, then on Duplicati, then running the backup.

That’s the Backup destination portion of the job’s Destination screen. There’s a Test connection button too. What happens when you press it, and how quickly does it error? If it’s still the timeout, netstat -an could show what’s going on, in terms of what communication is being attempted where, and how far it proceeds.

netstat

You could pipe that into find if it makes things easier to spot. I assume this is Windows, if you run PuTTY (which is not an SFTP client, but its relative PSFTP is, and testing all the way through sftp may be helpful).

You could also go to the job’s Commandline option to see if the Target URL looks reasonable. An example:

ssh://localhost:220/the-backup-name?auth-username=xxx&auth-password=yyyr&ssh-fingerprint=zzz

[Mugros]

It was actually my fault. There are two settings where the port 22 shows up and I changed the wrong one.
It works now.
Thanks.

[ts678]

I found new information, so here’s an update on the problem that @DL38 was having in the original post.

@ts678

Bottom line here is that if you can figure out how to fix moduli easily, try it

Maybe the later openssh fix below would have avoided the original problem:

No moduli file in C:\ProgramData\ssh or anywhere else I can find. #1261
Ship Moduli in ssh #350

I’m pleased to see this openssh still being worked on. Sometimes it’s handy to have a Windows openssh.

Though @Mugros had the same symptom but a different cause, I’m glad that issue is now resolved too.

[DL38]

Hello @ts678, thank you for continuing to follow this. I have read up on your links and would like to try the proposed moduli solution at Ship Moduli in ssh #350 but I am beyond my technical understanding.

I tried following the links and was able to generate a moduli file using PowerShell:

PS C:\Users\CompA> & ‘C:\Program Files\OpenSSH\ssh-keygen.exe’ -G moduli -b 4096
Sun Feb 24 18:12:50 2019 Sieve next 268304384 plus 4095-bit
Sun Feb 24 18:17:44 2019 Sieved with 203277289 small primes in 294 seconds
Sun Feb 24 21:22:00 2019 Found 221664 candidates

This file was 225MB and I dropped it into C:\ProgramData\ssh on the server (Comp A, the Win 7 64-bit computer). I restarted the OpenSSH service on server, and restarted Duplicati on the client (Comp B) side, and tested the connection from Duplicati, and there was a still a timeout. The Event Viewer log for the server reads:

sshd: WARNING: no suitable primes in PROGRAMDATA\ssh/moduli

So, I am guessing I’ve done something incorrect here with respect to the moduli file and haven’t been able to test what actually may be wrong.

Any tips are appreciated and thank you again for the continued help.