Http send report errors / duplicati-monitoring

My systems that’re having a fistful of these use 1.1.1 of OpenSSL. But if a lower version is embedded in Mono…

A similar thing happened, IIRC, when Heartbleed struck. People upgraded OpenSSL in their main software config, but oodles of other products had worse versions of OpenSSL libs linked in to their binaries.

Done a small bit of digging but dont have any more time today to look at.

Centos 8 Openssl 1.1.1. Updated ca-certificates and re synced the mono cert store

Ran Certmgr and noted that ISRG Root X1 Certificate Serial Number does not match the serial number when connected via browser.

After some quick search, it may be on server’s side also.
The server may cache the old certificates chain.
A reboot will purge the server’s cache.

I have the issue with one of my server, and I can’t connect only from my iphone.
I have issued a new certificate and I will reboot the server as soon as I can.
But from my Windows PC, I can connect to it… so it is strange…

To come back to duplicaty, the issue for me is when duplicati trie to send the report to the site www.duplicati-monitoring.com
It is a Let’s encrypt certificate too, but I don’t have any issue to connect to the site with my iPhone or Windows PC…
So maybe the site’s server has been rebooted ? (certificate has been issued on Sept 29)

So it may be on server’s side…

Got a resolve/workaround for Centos8 System.

Removed the expired DST Root CA X3 Cert from /etc/ssl/certs/ca-bundle.crt

Then cleared it from mono cert store
cert-sync /etc/ssl/certs/ca-bundle.crt

Unfortunately this may break when we do mono or ca-cert updates down the line.

I will try the same on Centos7 and Syno NAS later but openssl 1.0 may bite me here

Help thread for DST Root CA X3 expiration (September 2021) at Let’s Encrypt Community Support is extremely busy now, and too technical for me. Thanks to the better experts who are contributing here.

When the dust settles, easy step-by-step directions would be helpful, because some may want them.

I was having issues with this as well… not just with the http send report, but also with performing the backup itself because my backup destination is a WebDAV server that uses StartSSL.

On Debian 10, I was able to work around this problem by performing the following:

##!!! use at your own risk.. I do NOT suggest deleting the offending cert 
##!!! but instead moving it to another location in case you need to put 
##!!! things back the way they were for some reason.

# update apt
apt-get update

# make sure the latest ca certs are installed
apt-get install ca-certificates

# remove the offending root cert from the root store
mv /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt ~/DST_Root_CA_X3.crt 

# update the root ca store.. this automatically updated mono for me.
update-ca-certificates

After I performed the above steps (as root or with sudo), my backups started working again. Didn’t even need to restart duplicati.

This may come around again in the future, but for now at least I can backup my systems.

Hope this helps someone!

7 Likes

I don’t really understand how this works because the DST_Root_CA_X3 isn’t anywhere in the cert chain for the duplicati-monitoring site. But it does work! Thanks. I spent a few hours yesterday trying to fix this.

Glad it helped! I honestly don’t fully understand it either, but then again, I am in no way an encryption expert.

I just followed the notes provided by @ccmgr but translated it to something that worked on Debian 10. Thank them for the initial idea!

Yes that’s correct on a web browser but if you run monos certmgr -ssl -c https://duplicati-monitoring.com it will show DST_Root_CA_X3 in the chain.

I applied the same fix to a Centos 7 System but cert-sync did not totally remove the offending certs.
certmgr report the DST cert not installed but on future investigation physical cert files where in the store…
Had to take a brute force option
Mono certs stores are in .~/.config/.mono/ or /usr/share/.mono depending on setup
The folders may have certs in either or in my case both certs\Trust and new-certs\Trust. I had to grep to find the DST cert files and move/remove them.

You dont have to run a backup to test as checking for an update will throw the same error

2 Likes

Interesting… wonder why there is a discrepancy…

I’m glad people are finding a workaround. To me it looks worse than the symptom; I will leave things alone until a proper update works its way through to me.

But my backups run, so I can afford to wait.

Enable X509_V_FLAG_TRUSTED_FIRST flag in BoringSSL #21233 is the latest-updated mono issue.
I’m hoping staying current with mono right from mono-project will resolve. Distros may do fixes slower.

No sucess on Synlogy yet. Hoever can across this blog which outlines and explains the issue

I guess you are running the native Duplicati package on Synology? I use the docker image on my Synology and the workaround worked for me.

My backups run - my only impact is missing duplicati-monitoring reports. I can leave it be now.

yeah for my sins its a DS218J no docker support for a small client

I had the same issue, and the solution from @hairlesshobo worked for me.

Sorry for joining the party so late. Yes, the root cause is that we use a letsencrypt certificate that is automatically updated regularly and their new root cert is not cross-signed anymore.

But: If this certificate causes errors on your client, you should really update your client. It is not like the “new” ISRG Root X1 certificate is something that popped up last month. Most browsers and operating systems trust it since years. If your client does not, it probably is badly outdated.

In our statistics we see that we continue to receive more and more backup reports. Currently about 16000 per day. So the vast majority of clients does not seem to have a problem with the new certificate.

1 Like

I do agree with you @Kahomono. No time to spend with workarounds either, and backup are working fine, of course without monitoring :unamused:

@Kahomono and @Rajstopy
The workaround is the fix. See my post with centos link
Updating the ca-certificates remove the offending expired certificate from your systems.
Using certmgr to recreate a copy of the new certificate’s in mono’s store. I found in some cases certmgr did not remove the expired cert and I had to manually delete same cert.
It will also prevent you from doing an upgrade to duplicati as the site use a Lets Encrypt Certificate

1 Like