SFTP backup no longer working on Asustor NAS after patch

Maybe issues such as these?

X509Certificate creation issue in Xamarin.IOS for elliptical curve algorithm. (ECDSA) #21275

Cannot create ECDsa keys to access Apple services #18821

I’m kind of wondering if Asustor’s gap came from a dependency. OpenSSH changed libraries:

https://www.openssh.com/txt/release-9.4

Potentially incompatible changes

  • This release removes support for older versions of libcrypto.
    OpenSSH now requires LibreSSL >= 3.1.0 or OpenSSL >= 1.1.1.
    Note that these versions are already deprecated by their upstream
    vendors.

It’s not clear which OpenSSH Asustor had previously, but 9.4 was about two months before 9.5.

EDIT 1:

Can’t connect (SCP, SSH) in Xamarin Android #807
is a 2021 SSH.NET issue involving Duplicati and probably my May 2021 .pcap files on Windows.
Although it won’t help us near-term, I wonder if the .NET 5 (or any non-mono) port works better?

Oh well, I tried again with current Duplicati under Windows and it fails, I tried again with the native Linux client for Ubuntu 20.04 and it fails (while it connected before), so I have to assume that the OP Nas does not allow connections anymore.
For completeness, I installed Arch Linux in a container to get Openssh 9.5 and it works fine with current Duplicati (SSH.NET 2020.0). That’s the end of my involvement with this matter if the OP does not show further interest.

FWIW it still works here (meaning Test connection button gets Trust host certificate? blue box).

First of all thanks a lot for all your analysis and feedback. Really appreciate it!

if the OP does not show further interest

I apologize for any delay but I do have some other things to do in my life :wink: and I am most likely in another timezone than you (+7). I will try to answer asap but sometimes I can’t do it immediately.

For all the feedback regarding connection issues let me say that I have not changed anything. Did a quick test just now and I can still connect as shown above using FileZilla.

@ts678 Yes, I install whatever older version from AppStore and then click on “Search update” on the “About” page which downloads and installs the version I mentioned. That’s what I meant saying “That’s what it automatically downloads on Asustor” which was probably not that clear (sorry for that). But there is a caveat here: After restarting the NAS the new version is gone which requires me to update again after every NAS restart. Maybe I should open up another issue for that but until now it didn’t bother me too much because a NAS restart is something I do only a handful of times per year.

I connected to my Asustor using ssh and tried to look for sshd_config as suggested by @Jojo-1000. There are a number of files with that name on my volume, mostly in a subfolder of docker-ce (yes, I also have Docker installed from the AppStore in case that matters though I wouldn’t think so). The most interesting one seems to be the only non-Docker one in “/usr/etc/ssh” but there are no algorithms configured in there:

# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /usr/etc/ssh/ssh_host_rsa_key
HostKey /usr/etc/ssh/ssh_host_dsa_key
HostKey /usr/etc/ssh/ssh_host_ecdsa_key
#Privilege Separation is turned on for security
#UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 3600
#ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

#RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile     %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /usr/etc/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication yes

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /usr/etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /bin/false

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
UseDNS no

#Ciphers aes128-ctr,aes192-ctr,aes256-ctr
#MACs hmac-sha1,hmac-ripemd160

However, if I run “ssh -Q HostKeyAlgorithms” I get:

ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ssh-ed25519-cert-v01@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521
ecdsa-sha2-nistp521-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
ssh-dss
ssh-dss-cert-v01@openssh.com
ssh-rsa
ssh-rsa-cert-v01@openssh.com
rsa-sha2-256
rsa-sha2-256-cert-v01@openssh.com
rsa-sha2-512
rsa-sha2-512-cert-v01@openssh.com

So the algorithms you mentioned are active already - but maybe this goes for my current ssh session only and sftp sessions have a different config? I have no idea how to check (or change) that…

@ts678 I tried “systemctl status sshd” but got “systemctl: not found”. I looked in “/var/log” and /usr/var/log" but couldn’t see anything related to ssh, sftp, ssl etc. I searched for files ending with “.log” on the entire volume but again saw nothing relevant. However, I found the Duplicati folder and a log file in there which has a lot of entries saying “SshConnectionException: An established connection was aborted by the server.” including the full stack trace. This is not really new information but in case the full stack trace helps I can get it from there. I understand that the really interesting information would be the sftp error message on the server but I couldn’t find that yet. Asus is messing around with the linux setup and it can get hard to find out what their configuration works like…

Regarding Wireshark and other package dumps I must say that I see this as a last desperate effort and something I would like to avoid if possible. When going on that layer it usually becomes quite time consuming and complicated so let’s try to make use of all other means of investigation first. If someone has a clue where to find the sftp logs or configuration files on Asustor that would probably be a good starting point. I will also ask in the Asus forum although Asus support is usually completely useless.

I think I have addressed all feedback now, let me know if I missed something…

I did not either, so my thinking is that a firewall has locked my IP for excessive login attempts. I control other internet addresses so I set up a relay to make my connection come from another IP and then I can repro the same thing I said, that is, with current Duplicati there is a connection reset by peer, and with the sftp driver set to 2023.0 I get a password error. In this case the fingerprint is indeed set to rsa-sha2-512 2048.

Edit: 3 hours later, my new IP address don’t work anymore.
Edit 2: I see, that’s it:
https://www.asustor.com/en/online/online_help?id=9

I don’t think you have a current ssh session. Isn’t that run just asking the client for its configuration? Attempting to find an easy way to determine what the sshd uses led me first to a remote query way:

Listing host’s available ssh ciphers from client

ssh -vv -p 12345 184.82.24.166

debug2: peer server KEXINIT proposal
...
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256

so there’s that lost ssh-ed25519 again. I’m not seeing security hate for it, in fact I even found:

It’s 2023. You Should Be Using an Ed25519 SSH Key (And Other Current Best Practices)

Can Asustor be safely downgraded to see if it was there before? If so, that supports the theory.

Possibly it’s still a little premature, but it seems fair to ask Asustor why the algorithm isn’t there.
From your ssh client testing, it appears to be on the system – except not in their shipped sshd.

There are defaults. We’re probably seeing them. The referenced sshd_config man page says.

Ciphers
Specifies the ciphers allowed. Multiple ciphers must be comma-separated. If the specified list begins with a ‘+’ character, then the specified ciphers will be appended
to the default set instead of replacing them.

This plus method of querying the sshd should be enough to let you see if you can get lost cipher.
I’m not sure what the risk is if sshd breaks (and how to fix it), but your backups are already down.

OpenSSH Manual Pages links docs. I don’t know if Asustor is systemd-based. You could ps test.
sshd page says -E can log to a log_file, but I’m not sure how you’d configure that into Asustor.

EDIT 1:

Although we might trip over other settings later, the relevant setting to start with might be this one:

HostKeyAlgorithms
Specifies the host key signature algorithms that the server offers

and on the client side if one wants to drive the sshd in a certain way, the setting might be this one:

HostKeyAlgorithms
Specifies the host key signature algorithms that the client wants to use in order of preference. Alternately if the specified list begins with a ‘+’ character, then the specified signature algorithms will be appended to the default set instead of replacing them.

EDIT 2:

I notice that the client man page says:

The list of available signature algorithms may also be obtained using “ssh -Q HostKeyAlgorithms”.

which is sort of puzzling. I don’t think it can reach into an un-named connection or remote sshd, but maybe it can speak for the sshd on its own system. I don’t think you said what system you were on.

EDIT 3:

Because I don’t have an Asustor to try to fix, I tried breaking Manjaro using the following config line:

HostKeyAlgorithms rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256

and got its ssh to match Duplicati’s limited-by-what-mono-can-provide configuration using config line:

HostKeyAlgorithms ssh-ed25519,ssh-rsa,ssh-dss

systemctl restart sshd (known to not work on Asustor – it reportedly uses a different restart way).

$ ssh -vv 127.0.0.1
OpenSSH_9.5p1, OpenSSL 3.1.4 24 Oct 2023
...
debug1: Local version string SSH-2.0-OpenSSH_9.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.5
...
debug2: host key algorithms: ssh-ed25519,ssh-rsa,ssh-dss
...
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256
...
debug1: kex: host key algorithm: (no match)
Unable to negotiate with 127.0.0.1 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256
$ 

EDIT 4:

I was going to show a failure to the Asustor with my cut-down ssh client, but it looks different now:

debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256

so maybe workaround is already underway? Long-term question is whether ssh-rsa is the safest.

EDIT 5:

Probably OK for some test, but don’t stay on that one, at least according to information found here.

Yes, I think that it’s the problem.
I have lost hope of reliably connecting to the OP site, so I have used my test Arch install to make some variations in sshd_config; with current Duplicati, not having

HostKey /etc/ssh/ssh_host_ed25519_key

reproduces the problem with current trunk; with current trunk + 2023.0 driver, anything goes.

So @srudin can you try to add this stanza to your sshd_config file (order of directives don’t seem to matter), generate a key with something like

ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key

and restart the sshd service; if you don’t have systemd, it should be (hopefully)

/etc/init.d/sshd restart

look into the /etc/init.d directory to see if something looks like sshd if it does not find this file.

When looking to see if Asustor had systemctl, these answers say no, but suggest other restart ways:

Restart service?
Restart services via terminal plus cron job?

There might be other answers there. Answering how to operate an Asustor is getting out of scope here.

I have been able to get it to work. It’s pretty late here already so please allow me to post all details tomorrow.

1 Like

You have all been on the right track!

I got a hint in the Asus forum post I mentioned at the beginning. Seems that there are 2 sftp servers on Asustor called “administrative sftp” and “user sftp” and they have different config files. The active one on my machine was the “user sftp” and its config is located at /usr/builtin/etc.default/sshd_config_sftp (while the config of the “administrative sftp” is located at /usr/etc/ssh/sshd_config). However, both configs did not contain any HostKeyAlgorithms or PubkeyAcceptedKeyTypes so I assume they just inherit some defaults.

Knowing this Asustor specific setup I could now add these lines at the end of the correct config file:

HostKeyAlgorithms +ssh-rsa
PubkeyAcceptedKeyTypes +ssh-rsa

This is pretty much what you all suggested. I also tried it with ssh-ed25519 but that didn’t work, maybe that one was entirely removed from the machine or whatever. I can live with ssh-rsa for now. After adding this and restarting the service it worked.

So thanks again to all of you - you nailed it quickly!

The question remaining is why Asus has changed this configuration and what the proper long term solution is. From what I understand Asus has updated a library and thinks that this increases security by disabling old crypto algorithms. On the other hand Duplicati is using pretty up to date libraries on the client side yet they can no longer talk with each other. So it seems to me that either one side is not really up to date or someone has made a mistake. I mean if the defaults from the newest OpenSSH and SSH.NET versions were not compatible, someone would/should have noticed already, no?

Finding a solution would be helpful because changing the config files on Asustor is not something that should be necessary to do (and probably not everyone can do it).

Btw I have now disabled the above described access to my NAS.

Where is this from? The Asustor release note just cites AS-2023-013: OpenSSH and CVE-2023-38408. Algorithms do sometimes get disabled, but did you find any concern about ed25519? Looked OK to me.

It’s not just the SSH library. Just as apps might not roll their own SSH, SSH library might get crypto from somewhere else. Apparently, for the ecdsa protocols, that’s from the interpreter running the actual code.

The mono runtime aims for .NET Framework compatibility on non-Windows platforms, but both products are kind of stagnant, as Microsoft bought Xamarin (mono creators), but probably not to improve mono.

See Introducing .NET 5 and how they intend a single unified cross-OS platform for uses going forward.

I linked to some mono issues about lack of ecdsa. I’m not seeing similar ones for sha2, and there was a test of it against your pre-modification server, so perhaps that’s another client solution to explore further:

Sounds reasonable, though further testing might be in order. Is this a drop-in replacement DLL?
We’ve had a few users willing to replace those, instead of waiting for a Canary, or the later Beta.

Newest OpenSSH is not that widely rolled out. You can see what distro has what at repology.org.
That’s why I tested on Manjaro (an Arch derivative) and there was an independent test with Arch.
In addition to testing the scenario you mention (and finding it compatible but not on your Asustor), checking for such issue in the SSH.NET issues wasn’t seeing this, as it’s maybe Asustor-specific.

We don’t know the Asustor OpenSSH history. It’s not guaranteed they broke this by removing the ed25519 protocols intentionally. It could have been a result of how they built it that slipped by QA. Perhaps they took a dislike to ed25519 long ago for some reason, and dropped some other aging algorithm such as ssh-rsa in their latest release, thereby pulling out the last compatible algorithm.

I expect Asustor has a system with their previous release somewhere, but will they test it for you?
They don’t need to install Duplicati. The ssh -vv test should be enough to see what sshd allows.

It is also relevant which host keys exist on the server. The possible variants are RSA DSA ECDSA ED25519. Maybe only RSA and ECDSA were used when the system was originally created, so now even if ed25519 is allowed by default, it has no key so it can’t be used.

That is why @gpatel-fr suggested the keygen command, but if it works now you should be good.

We should look into updating SSH.NET, because ssh-rsa is deprecated since 2021 in favor of rsa-sha2 (uses SHA2 instead of 128-bit SHA1).

https://www.openssh.com/txt/release-8.8

This release disables RSA signatures using the SHA-1 hash algorithm
by default. This change has been made as the SHA-1 hash algorithm is
cryptographically broken, and it is possible to create chosen-prefix
hash collisions for <USD$50K [1]

For most users, this change should be invisible and there is
no need to replace ssh-rsa keys. OpenSSH has supported RFC8332
RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys
will automatically use the stronger algorithm where possible.

Good point.

When I was first fumbling around with this, I found a post suggesting ssh-keygen -A to make keys:

# pwd
/etc/ssh
# ls *ecdsa*
ssh_host_ecdsa_key  ssh_host_ecdsa_key.pub
# rm *ecdsa*
# ssh-keygen -A
ssh-keygen: generating new host keys: ECDSA 
# 

With nonstandard Asustor folders, maybe avoiding hardcoded paths would hit the correct locations?
I suppose a missing-key theory could be another explanation for why ed25519 stubbornly won’t run.

If this is the case (but it would be best to look in the relevant folders now that they’re better known):

Shodan boss finds 250,000 routers have common keys was a big goof on an image-based system.
I don’t know how Asustor does updates, but I have a feeling it’s as-a-whole not separate packages.

Yes it is quite possible, I did a quick test with 2023.0 and the upgrade just works, I’m a bit worried with compatibility with old SSH servers.
IIRC there is a TODO in the 2023.0 upgraded code about that, seems there is no setting to allow to work with legacy servers yet. I have not looked at this problem closely, if there is really a security problem upgrade is forced anyway and it’s just too bad for legacy stuff, but if there is not I’m not as keen. At the moment I don’t know.

I think they disabled ssh-rsa. And that one is officially deprecated.

That’s the problem with Asus. I like their hardware but they do deserve the golden raspberry award for their software: It’s user unfriendly, constantly buggy, badly documented and completely unsupported - which also answers your question about getting support from their side. So no, most likely we won’t be able to find out what they really intended nor will they be of any help. At least I won’t waste my time trying, did it too many times already.

So I guess what it comes down to is: Is Duplicati going to be upgraded with an SSH.NET version that supports any of the ssh-rsa2 algorithms? That would save the problem for everyone and allow me to rollback my manual changes without the need of Asus getting involved.

The interim maintainer said (right above your post):

I’m pretty sure it will happen eventually, but it’s a major release one month ago with a lot of changes.
Their 2020 release went through four versions. Are you sure you want to be on the first try for 2023?

Duplicati issues prereleases through the Canary channel. Would you be willing to switch in Settings? Canary releases can be very raw. Usually a Beta comes from a series of Canary releases, plus time.
Potentially risky changes don’t get in late in the series of Canary releases. They need more test time.

I don’t know the plan for next Beta. It’s always a tradeoff between getting more in versus fast release.
By historical standards, this is way too early for a new Beta, but maybe a little slow for a new Canary.

seems like it has some people wanting a Canary, and there are some details. Would you go Canary?
The 2023 Beta was about two years after 2021 Beta due to staff shortages. Usually it’s under a year.

Did you do the requested check to make sure you have an ed25519 key? A corrected example of this:

# pwd
/etc/ssh
# ls -l1 *key*
-rw------- 1 root root  525 Nov 11 08:34 ssh_host_ecdsa_key
-rw-r--r-- 1 root root  189 Nov 11 08:34 ssh_host_ecdsa_key.pub
-rw------- 1 root root  419 Nov 10 09:53 ssh_host_ed25519_key
-rw-r--r-- 1 root root  109 Nov 10 09:53 ssh_host_ed25519_key.pub
-rw------- 1 root root 2622 Nov  8 22:01 ssh_host_rsa_key
-rw-r--r-- 1 root root  581 Nov  8 22:01 ssh_host_rsa_key.pub
# rm *ed25519*
# ls -l1 *key*
-rw------- 1 root root  525 Nov 11 08:34 ssh_host_ecdsa_key
-rw-r--r-- 1 root root  189 Nov 11 08:34 ssh_host_ecdsa_key.pub
-rw------- 1 root root 2622 Nov  8 22:01 ssh_host_rsa_key
-rw-r--r-- 1 root root  581 Nov  8 22:01 ssh_host_rsa_key.pub
# ssh-keygen -A
ssh-keygen: generating new host keys: ED25519 
# ls -l1 *key*
-rw------- 1 root root  525 Nov 11 08:34 ssh_host_ecdsa_key
-rw-r--r-- 1 root root  189 Nov 11 08:34 ssh_host_ecdsa_key.pub
-rw------- 1 root root  419 Nov 12 08:39 ssh_host_ed25519_key
-rw-r--r-- 1 root root  109 Nov 12 08:39 ssh_host_ed25519_key.pub
-rw------- 1 root root 2622 Nov  8 22:01 ssh_host_rsa_key
-rw-r--r-- 1 root root  581 Nov  8 22:01 ssh_host_rsa_key.pub
# 

Now that you have some information on the folder locations (I think), could you please look at keys?
That’s by far the easiest and safest solution if it works. If not, maybe Canary, or drop in a new .dll?

Was that a drop in .dll pulled from NuGet? If it didn’t need a new build, maybe @srudin could try?
We might have to find Asustor Duplicati update’s location and solve disappearance problem though.
Switching to Canary is probably no worse than the current Beta glitch, but Canary is itself a bit risky.

No, it was a private rebuild.

Hm, I’m a little reluctant to switch to Canary. I’m quite busy and don’t want to ruin my backups because I don’t have the time to setup everything again. I think I’ll wait for a new Beta - then I’m willing to invest some time, remove the workaround and try if it works without. Let me know when that’s the case. Meanwhile I’ll stick to the workaround.

We might have to find Asustor Duplicati update’s location and solve disappearance problem

That would be very helpful!

make sure you have an ed25519 key

Thanks, I understand this would be slightly better but I’ll stick to ssh-rsa. Everything is working now and as said I don’t have a lot of time for this kind of work.

Thanks again for all your support! Very much appreciated.

When Duplicati notifies you of the next Beta, you can see changes in at least one of these places:

Forum Releases category which can also accept feedback, in case there were unexpected issues.
https://github.com/duplicati/duplicati/releases

System-specific issues may need assistance of the person with that system, so get back sometime.