Anyone else seeing errors like this AFTER the backup is successful? I send my reports to duplicati-monitoring.com and this began happening to every backup job sometime after 9:30 AM EDT yesterday
2021-09-30 20:40:32 -04 - [Warning-Duplicati.Library.Modules.Builtin.ReportHelper-ReportSubmitError]:
Failed to send message: System.Net.WebException:
Error: TrustFailure (Authentication failed, see inner exception.) -->
System.Security.Authentication.AuthenticationException:
Authentication failed, see inner exception. -->
Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL
routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at
/build/mono-6.12.0.122/external/boringssl/ssl/handshake_client.c:1132
Getting the same error on One of my backups as well. I note the certificate was changed on 29/09/2021 on the website. I didn’t get this error on all backups I have a mix of Centos and Windows so I will try to narrow down the issue
I’ve just spent 3 hours trying to understand why this happens , since everything worked like a charm yesterday, I didn’t change anything because I was just sleeping, and today, my 3 Duplicati instances are no more able to send any e-mail.
I use a Synology mail server (like ccmgr), and the certificate has been renewed last week-end. My other servers do send notification without problem as well.
Here is the error message I get:
MailKit.Security.SslHandshakeException: An error occurred while attempting to establish an SSL or TLS connection. The SSL certificate presented by the server is not trusted by the system for one or more of the following reasons: 1. The server is using a self-signed certificate which cannot be verified. 2. The local system is missing a Root or Intermediate certificate needed to verify the server's certificate. 3. The certificate presented by the server is expired or invalid. See https://github.com/jstedfast/MailKit/blob/master/FAQ.md#InvalidSslCertificate for possible solutions. --> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. --> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at /build/mono-5.20.1.34/external/boringssl/ssl/handshake_client.c:1132
I did suspect my Synology first since all Duplicati instances report the same message. First checks using openssl s_client don’t show up anything abnormal either…
This goes further then Duplicati Monitoring. I updated Mono tp 6.12.107 via yum on Centos8 System. Then I tried to update Duplicati from 2.6.01_Beta to 2.6.03 and I got the exact same error when trying to download the newer version from the web ui.
DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates.
So maybe it is on client side the fix needs to be done ?
…
What should you do? For most people, nothing at all! We’ve set up our certificate issuance so your web site will do the right thing in most cases, favoring broad compatibility. If you provide an API or have to support IoT devices, you’ll need to make sure of two things: (1) all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and (2) if clients of your API are using OpenSSL, they must use version 1.1.0 or later.
My systems that’re having a fistful of these use 1.1.1 of OpenSSL. But if a lower version is embedded in Mono…
A similar thing happened, IIRC, when Heartbleed struck. People upgraded OpenSSL in their main software config, but oodles of other products had worse versions of OpenSSL libs linked in to their binaries.
After some quick search, it may be on server’s side also.
The server may cache the old certificates chain.
A reboot will purge the server’s cache.
I have the issue with one of my server, and I can’t connect only from my iphone.
I have issued a new certificate and I will reboot the server as soon as I can.
But from my Windows PC, I can connect to it… so it is strange…
To come back to duplicaty, the issue for me is when duplicati trie to send the report to the site www.duplicati-monitoring.com
It is a Let’s encrypt certificate too, but I don’t have any issue to connect to the site with my iPhone or Windows PC…
So maybe the site’s server has been rebooted ? (certificate has been issued on Sept 29)
I was having issues with this as well… not just with the http send report, but also with performing the backup itself because my backup destination is a WebDAV server that uses StartSSL.
On Debian 10, I was able to work around this problem by performing the following:
##!!! use at your own risk.. I do NOT suggest deleting the offending cert
##!!! but instead moving it to another location in case you need to put
##!!! things back the way they were for some reason.
# update apt
apt-get update
# make sure the latest ca certs are installed
apt-get install ca-certificates
# remove the offending root cert from the root store
mv /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt ~/DST_Root_CA_X3.crt
# update the root ca store.. this automatically updated mono for me.
update-ca-certificates
After I performed the above steps (as root or with sudo), my backups started working again. Didn’t even need to restart duplicati.
This may come around again in the future, but for now at least I can backup my systems.
I don’t really understand how this works because the DST_Root_CA_X3 isn’t anywhere in the cert chain for the duplicati-monitoring site. But it does work! Thanks. I spent a few hours yesterday trying to fix this.
Yes that’s correct on a web browser but if you run monos certmgr -ssl -c https://duplicati-monitoring.com it will show DST_Root_CA_X3 in the chain.
I applied the same fix to a Centos 7 System but cert-sync did not totally remove the offending certs.
certmgr report the DST cert not installed but on future investigation physical cert files where in the store…
Had to take a brute force option
Mono certs stores are in .~/.config/.mono/ or /usr/share/.mono depending on setup
The folders may have certs in either or in my case both certs\Trust and new-certs\Trust. I had to grep to find the DST cert files and move/remove them.
You dont have to run a backup to test as checking for an update will throw the same error
, since everything worked like a charm yesterday, I didn’t change anything because I was just sleeping, and today, my 3 Duplicati instances are no more able to send any e-mail.