Http send report errors / duplicati-monitoring

Anyone else seeing errors like this AFTER the backup is successful? I send my reports to duplicati-monitoring.com and this began happening to every backup job sometime after 9:30 AM EDT yesterday

2021-09-30 20:40:32 -04 - [Warning-Duplicati.Library.Modules.Builtin.ReportHelper-ReportSubmitError]:
Failed to send message: System.Net.WebException: 
Error: TrustFailure (Authentication failed, see inner exception.) --> 
System.Security.Authentication.AuthenticationException: 
Authentication failed, see inner exception. --> 
Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL 
routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at 
/build/mono-6.12.0.122/external/boringssl/ssl/handshake_client.c:1132

No changes to these configurations in weeks.

Getting the same error on One of my backups as well. I note the certificate was changed on 29/09/2021 on the website. I didn’t get this error on all backups I have a mix of Centos and Windows so I will try to narrow down the issue

Update

This is only happening on my Centos install as well as the one Sinology NAS I have it on.

These all different versions of Mono but I suspect Mono doesn’t like the new certificate that is on the site now

+1 !

I’ve just spent 3 hours trying to understand why this happens :rage: , since everything worked like a charm yesterday, I didn’t change anything because I was just sleeping, and today, my 3 Duplicati instances are no more able to send any e-mail.

I use a Synology mail server (like ccmgr), and the certificate has been renewed last week-end. My other servers do send notification without problem as well.

Here is the error message I get:

MailKit.Security.SslHandshakeException: An error occurred while attempting to establish an SSL or TLS connection. The SSL certificate presented by the server is not trusted by the system for one or more of the following reasons: 1. The server is using a self-signed certificate which cannot be verified. 2. The local system is missing a Root or Intermediate certificate needed to verify the server's certificate. 3. The certificate presented by the server is expired or invalid. See https://github.com/jstedfast/MailKit/blob/master/FAQ.md#InvalidSslCertificate for possible solutions. --> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. --> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at /build/mono-5.20.1.34/external/boringssl/ssl/handshake_client.c:1132

I did suspect my Synology first since all Duplicati instances report the same message. First checks using openssl s_client don’t show up anything abnormal either…

So far I’m stuck

Hello,
Maybe it is related to this ?
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

I had the same error this morning. I will try to update the docker container I use for duplicati

regards

2 Likes

This goes further then Duplicati Monitoring. I updated Mono tp 6.12.107 via yum on Centos8 System. Then I tried to update Duplicati from 2.6.01_Beta to 2.6.03 and I got the exact same error when trying to download the newer version from the web ui.

Update:

May be linked…

The error messages are pretty clear I think :

And me I have : routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Wondering if @crazy4chrissi can fill us in if a fix for this situation is coming from the server end?

taken from https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates.

So maybe it is on client side the fix needs to be done ?
…
What should you do? For most people, nothing at all! We’ve set up our certificate issuance so your web site will do the right thing in most cases, favoring broad compatibility. If you provide an API or have to support IoT devices, you’ll need to make sure of two things: (1) all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and (2) if clients of your API are using OpenSSL, they must use version 1.1.0 or later.

My systems that’re having a fistful of these use 1.1.1 of OpenSSL. But if a lower version is embedded in Mono…

A similar thing happened, IIRC, when Heartbleed struck. People upgraded OpenSSL in their main software config, but oodles of other products had worse versions of OpenSSL libs linked in to their binaries.

Done a small bit of digging but dont have any more time today to look at.

Centos 8 Openssl 1.1.1. Updated ca-certificates and re synced the mono cert store

Ran Certmgr and noted that ISRG Root X1 Certificate Serial Number does not match the serial number when connected via browser.

After some quick search, it may be on server’s side also.
The server may cache the old certificates chain.
A reboot will purge the server’s cache.

I have the issue with one of my server, and I can’t connect only from my iphone.
I have issued a new certificate and I will reboot the server as soon as I can.
But from my Windows PC, I can connect to it… so it is strange…

To come back to duplicaty, the issue for me is when duplicati trie to send the report to the site www.duplicati-monitoring.com
It is a Let’s encrypt certificate too, but I don’t have any issue to connect to the site with my iPhone or Windows PC…
So maybe the site’s server has been rebooted ? (certificate has been issued on Sept 29)

So it may be on server’s side…

Got a resolve/workaround for Centos8 System.

Removed the expired DST Root CA X3 Cert from /etc/ssl/certs/ca-bundle.crt

Then cleared it from mono cert store
cert-sync /etc/ssl/certs/ca-bundle.crt

Unfortunately this may break when we do mono or ca-cert updates down the line.

I will try the same on Centos7 and Syno NAS later but openssl 1.0 may bite me here

Help thread for DST Root CA X3 expiration (September 2021) at Let’s Encrypt Community Support is extremely busy now, and too technical for me. Thanks to the better experts who are contributing here.

When the dust settles, easy step-by-step directions would be helpful, because some may want them.

I was having issues with this as well… not just with the http send report, but also with performing the backup itself because my backup destination is a WebDAV server that uses StartSSL.

On Debian 10, I was able to work around this problem by performing the following:

##!!! use at your own risk.. I do NOT suggest deleting the offending cert 
##!!! but instead moving it to another location in case you need to put 
##!!! things back the way they were for some reason.

# update apt
apt-get update

# make sure the latest ca certs are installed
apt-get install ca-certificates

# remove the offending root cert from the root store
mv /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt ~/DST_Root_CA_X3.crt 

# update the root ca store.. this automatically updated mono for me.
update-ca-certificates

After I performed the above steps (as root or with sudo), my backups started working again. Didn’t even need to restart duplicati.

This may come around again in the future, but for now at least I can backup my systems.

Hope this helps someone!

4 Likes

I don’t really understand how this works because the DST_Root_CA_X3 isn’t anywhere in the cert chain for the duplicati-monitoring site. But it does work! Thanks. I spent a few hours yesterday trying to fix this.

Glad it helped! I honestly don’t fully understand it either, but then again, I am in no way an encryption expert.

I just followed the notes provided by @ccmgr but translated it to something that worked on Debian 10. Thank them for the initial idea!

Yes that’s correct on a web browser but if you run monos certmgr -ssl -c https://duplicati-monitoring.com it will show DST_Root_CA_X3 in the chain.

I applied the same fix to a Centos 7 System but cert-sync did not totally remove the offending certs.
certmgr report the DST cert not installed but on future investigation physical cert files where in the store…
Had to take a brute force option
Mono certs stores are in .~/.config/.mono/ or /usr/share/.mono depending on setup
The folders may have certs in either or in my case both certs\Trust and new-certs\Trust. I had to grep to find the DST cert files and move/remove them.

You dont have to run a backup to test as checking for an update will throw the same error

2 Likes

Interesting… wonder why there is a discrepancy…