Baaahhh... authentication failing after upgrading ubuntu

I upgraded Ubuntu to 22.04. Everything went pretty seemless, but for two problems that are PROBABLY (?) related.

  1. Suddenly I couldn’t ssh to ubuntu machine from my windows desktop. Not sure why, since nothing changed. But a quick google of the error showed someone suggesting to update Putty on the Windows machine. I checked and I was using 0.72 and current was 0.77 so I updated and wouldn’t you know… it fixed it.

So I’m hoping that gives you some hint for…

  1. Duplicati. I get an error when trying to run any backup. The error is “Failed to connect: Permission denied (publickey)”

Coincidence?

Does Duplicati have to update something with regard to how it implements SSH? I’ve updated Duplicati on my machine to latest so I’m wondering if there’s something you guys have to do to bring it up to compliance or something like that?

Try to install Openssl 1.1.1 libraries. Duplicati don’t support Openssl 3, and out of the box it’s the only version supported by Ubuntu 22.04.

Thanks. That’s really unsafe advice, but after researching it seems like the only short-term solution.

I’m confused why Duplicati needs to “support” any given protocol at all. I’d think it would just call ssh that’s installed on the system and let it do its thing.

Anyway, my problem was 2-fold, it seems. Installing the 1.1.1 libraries didn’t help (at first). I also had to re-enable password login because even with the new libraries, it won’t connect with a key pair. With 1.1.1 installed (unsafe) and PasswordAuthentication set to yes (also… less… safe), I’m now able to resume backup (for safety lol). I’ve completely compromised my system, but hey… I can recover :slight_smile:

Like most software, Duplicati don’t call executables but libraries. A program can do its thing by calling another executable but it’s not the more reliable interface. Actually it’s less secure. So Duplicati is calling the library it supports and it happens to be missing thanks to an arbitrary decision by Ubuntu (that caused lot of problems to all kinds of other software)
I agree that it is annoying to have to use a PPA, but, hey, 20.04 is still good to go until 2025.

IDK about that. Seems strange. Maybe try to recreate a new key ?

It isn’t like there wasn’t a heads up on the arbitrary decision. But I get it, this is a dead project due to lack of volunteer developers. Considering how critical proper backups are, I’m shocked that there isn’t a well maintained offering, but whatever. I can’t help, so I’ll shut my mouth on that.

As for recreating a new key… I suppose I could try that, but the existing key works just not with Duplicati. It also didn’t work with Putty 0.72, as I mentioned, but once I updated to 0.77 it did without having to change the key.

New key didn’t help. But its OK. This system isn’t mission critical, so I’m OK with the security compromise. It works now, which is good enough.

actually there is quite a lot of unmerged PR but the few maintainers have currently other things to do. It’s not dead yet :slight_smile:

As the openssl 3 related issue on GitHub brings up, it may just be an issue with the 3rd party ssl project they use.

But, it might just be errors possibilities never considered.

Either way, Duplicati seems to have its own unique openssl issues.

Not using it with Duplicati seems to be the most practical thing to do. It you use it, it could break again at anytime. Though LTS will be best guarantee.

It might be better to grab a NAS drive with remote ssh if you need more security over PasswordAuthentication. Or something similar. I don’t need it as I only use ssh locally and Duplicati works fine on Ubuntu 22.04 without anything else. Upsides and downsides to everything.

NAS != backup.

For something to be suitably backed up, it needs to be offsite. That’s what I use Duplicati for, to back up my NAS to my friend’s server (and his to mine in exchange).

I’m glad backups are going again, but this is still pretty mysterious and not really satisfying.

Did you also test password authentication without 1.1.1 just to prove that both are needed?

Did friend upgrade remote similarly? Do you have ability to check OS and OpenSSH version?

Above has error similar to yours, and refers to this library issue:

whose status is closed, but I think a release has not yet been made with the fix. They lack time too.

https://repology.org/project/openssh/versions

says Ubuntu 22.04 has OpenSSH 8.9p1, which presumably kept above compatibility issue from 8.8.

Compared to 9 results for 8.7, the following Google search has 1050 for 8.8. Maybe more info there.

“permission denied publickey” “openssh 8.8”

So you use Ubuntu for the NAS? I’m confused by what you’re saying now lol. I was talking more specifically about NAS from large companies that may offer better remote access. You could then gain access to the files that way instead more securely for as long as it receives security updates. That should or might workaround the Ubuntu Openssl 3 issue if it exists.

Actually, I’m conflicted on it even being an Openssl Duplicati issue anyway. It might just be an issue with the 3rd party code that wasn’t noticed until Openssl 3. Or it might be a red herring. But, ssh use varies so it depends.

Don’t get into an argument with me about NAS != backup. I don’t want to hear it rofl :slight_smile:

Matter of fact I’m done here but that’s because I have no more time to spend on it.

No, now that you mention it. I had the initial publickey failure and came here and read about the 1.1.1 vs 3 thing and updated that first only to find it didn’t resolve the issue, then turned on passwords and the issue was resolved. I never did test without 1.1.1 and passwords on.

that’s a good question, and yes. I maintain his server too and I generally keep them in the same state at all times. His OS and OpenSSH version would be identical to mine.

I’m not sure what you mean by “for the NAS”. If you’re asking if my NAS runs Ubuntu, then no. My server runs Ubuntu and on the network is a NAS that stores my files locally. Sort of irrelevant whether it is a harddrive on my server or a NAS.

Are you meaning cloud backup?

Yes, you do seem confused.

Yeah, that’s fine.

It would be interesting to do the test by uninstalling the library.

Possibly also look for this, from the issue. I don’t know if logging is always enabled:

On the destination machine, the following is logged for the sshd daemon:

userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms

That would imply that the problem was the known incompatibility with OpenSSH 8.8.

The OpenSSL 3 connection was tempting. It does break some other Duplicati things:

which is a genuine problem, but I’m not sure if it’s the explanation for your SSH pains.
Also recall that PuTTY broke somehow, so an sshd might be the common connection.

Duplicati runs on many systems. Not all have ssh. Those that do may have differences.
Duplicati is mostly portable C# code, and SSH.NET is too. This aids cross-platform use.

Portable code generally avoids native libraries needs, so OpenSSL uses would be rare.
An exception is the FasterHashing library from the above-cited problem. It goes fishing:

Wrapper library for using native hashing libraries on Windows, Linux and OSX

Had this exact same problem after upgrading my storage server to Ubuntu 22.04.
I was able to get it to work without openssl 1.1.1 by using a ed25519 key.

Concerning “duplicati with openssl3”, you can try my solution here on github. This should fix openssl3 issues in FasterHashing.
++umgfoin.

Thanks for your work on this. I tried replacing on Windows 11 and am still getting the error. Have any ideas?

appears to have been proven. Please keep in mind that there are two issues intermixing here:

OpenSSL 3 broke FasterHashing and may need workaround. OpenSSH broke authentication,
however it was done in the name of improving security, as the following release note explains:

https://www.openssh.com/txt/release-8.8

This release disables RSA signatures using the SHA-1 hash algorithm
by default. This change has been made as the SHA-1 hash algorithm is
cryptographically broken, and it is possible to create chosen-prefix
hash collisions for <USD$50K

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy
implementations can be upgraded or reconfigured with another key type
(such as ECDSA or Ed25519).

In ssh-keygen, see option -t dsa | ecdsa | ecdsa-sk | ed25519 | ed25519-sk | rsa
If using something else, you can investigate what newer key types they have that SSH.NET has.

SSH.NET ECDSA support seems to be evolving. ed25519 looks like a safer bet if you can get it.
Public Key Authentication describes what it has. develop branch has others, but isn’t available.

The SSH.NET site is a better place to ask about their issues if the above doesn’t help. Examples:

Authentication exception when connecting to Fedora using private keys #834
SSH Key Authentication Fails with macOS Ventura #1003
Permission denied (publickey) with SSH.NET - Can connect via FileZilla or Powershell. #956

Jammy Jellyfish Release Notes
See Security Improvements section paragraph talking about ssh-rsa and workaround option.

If anybody else on this thread would like to say what solved the authentication issue, please help.

EDIT:

“openssh 8.8” “ssh-rsa” in a Google or similar web search can find a lot of commentary about that.

Keep in mind which side you’ve upgraded. If Destination end, OpenSSH may break authentication.
If upgrading system that Duplicati is on, OpenSSL 3 may break hashing, but that’s a different topic.

1 Like