Http send report errors / duplicati-monitoring

I had the same issue, and the solution from @hairlesshobo worked for me.

Sorry for joining the party so late. Yes, the root cause is that we use a letsencrypt certificate that is automatically updated regularly and their new root cert is not cross-signed anymore.

But: If this certificate causes errors on your client, you should really update your client. It is not like the “new” ISRG Root X1 certificate is something that popped up last month. Most browsers and operating systems trust it since years. If your client does not, it probably is badly outdated.

In our statistics we see that we continue to receive more and more backup reports. Currently about 16000 per day. So the vast majority of clients does not seem to have a problem with the new certificate.

1 Like

I do agree with you @Kahomono. No time to spend with workarounds either, and backup are working fine, of course without monitoring :unamused:

@Kahomono and @Rajstopy
The workaround is the fix. See my post with centos link
Updating the ca-certificates remove the offending expired certificate from your systems.
Using certmgr to recreate a copy of the new certificate’s in mono’s store. I found in some cases certmgr did not remove the expired cert and I had to manually delete same cert.
It will also prevent you from doing an upgrade to duplicati as the site use a Lets Encrypt Certificate

1 Like

Unfortunately, my mail server runs on Synology …

TL;DR I “think” this is a poor interaction between Let’s Encrypt default cert chain, OS certs, and mono.
This leaves multiple ways to fix, but the controllable one (though limited) is the certs that server sends.

I’m not a certificate wizard (but learning some), but the default one appears to be the cross signed one.
Wireshark shows that an Edge access to https://www.duplicati-monitoring.com gets the below:

Certificate: 3082056030820448a00302010202104001772137d4e942b8ee76aa3c640ab7300d06092a… (id-at-commonName=ISRG Root X1,id-at-organizationName=Internet Security Research Group,id-at-countryName=US)
    signedCertificate
        version: v3 (2)
        serialNumber: 0x4001772137d4e942b8ee76aa3c640ab7
        signature (sha256WithRSAEncryption)
            Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
        issuer: rdnSequence (0)
            rdnSequence: 2 items (id-at-commonName=DST Root CA X3,id-at-organizationName=Digital Signature Trust Co.)
                RDNSequence item: 1 item (id-at-organizationName=Digital Signature Trust Co.)
                RDNSequence item: 1 item (id-at-commonName=DST Root CA X3)
        validity
            notBefore: utcTime (0)
            notAfter: utcTime (0)
        subject: rdnSequence (0)
        subjectPublicKeyInfo
        extensions: 7 items
    algorithmIdentifier (sha256WithRSAEncryption)
    Padding: 0
    encrypted: 0a73006c966eff0e52d0aedd8ce75a06ad2fa8e38fbfc90a031550c2e56c42bb6f9bf4b4…

Note that the issuer is DST ROOT CA X3. The two versions of their certificate are described on page
Extending Android Device Compatibility for Let’s Encrypt Certificates showing default is cross-signed.
DST Root CA X3 Expiration (September 2021) talks about an Android exception. It doesn’t care about
DST Root CA X3 being expired. Other implementations do, such as older OpenSSL, and BoringSSL, which forked from it in 2014. mono uses BoringSSL. See the (oddly quiet) issue on it I mention earlier.

https://usage-reporter.duplicati.com/ suggests vast majority of Duplicati users use Windows not mono.

Resources for Certificate Chaining Help has a Workarounds for OpenSSL 1.0.2. link with three options.

1 is to remove the expired certificate as some are doing here, but I’m not sure how to keep it removed.
Ubuntu seems to have that problem solved by recently removing it from their ca-certificates packages.

2 is to ask the client to prefer the trust store certificates over the chain that the server sends. Without it, presumably the cross-sign of DST ROOT CA X3 is found and (except on Android) the access is failed.
The mono issue I cite asks them to do the change by using the X509_V_FLAG_TRUSTED_FIRST flag.

3 is for web server to get and use the alternative certificate chain that doesn’t go to DST ROOT CA X3.
It simply ends at ISRG Root X1 (self-signed) which presumably is trusted by most systems these days.

This breaks old Android, but that might not matter, and it probably solves the failures with current mono.
The limitation of this is that the server-side Let’s Encrypt workaround might need to be done elsewhere.
The procedure for asking for the alternative chain seemingly varies by ACME client (if it can even do it).

1 Like

For anyone running the native Duplicati package on Synology, I’ve documented how I solved it here (very easy, though it took huge amounts of flailing to get there). In brief:

sudo /volume1/@appstore/mono/bin/cert-sync \
    /etc/ssl/certs/ca-certificates.crt

… and restart Duplicati. More details at the link.

1 Like

@conrad Thanks for that. I will try later on system in question. I tried to update the certs using mozroots and also removing the offending cert from crt file but maybe the new cert wasn’t in it.

I fixed the cert issue on another system by Upgrading to DSM 7 but it broke my Duplicati and I havent tried to fix it yet, so reluctant to do the same on the other system

Mozilla’s CA Certificate Program leads to
Mozilla Included CA Certificate List shows ISRG Root X1 has been included for about 5 years.
DST Root CA X3 is also there, even though it’s expired. Remember that Android doesn’t care.

To further make the case that Duplicati Monitoring should favor current mono over old Android:

FAQ: Security at Mono Project suggests a test like:

csharp -e ‘new System.Net.WebClient ().DownloadString (“https://www.nuget.org”)’

I’ll also quote from result of openssl s_client -connect <host>:443 to show the certificate chain:

PASS at https://api.backblazeb2.com which apparently sacrificed old Android, thankfully for us:

Certificate chain
 0 s:CN = backblazeb2.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

PASS at https://valid-isrgrootx1.letsencrypt.org which is a test site for the new alternative chain:

Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

FAIL at https://www.duplicati-monitoring.com presumably because default chain does old Android:

$ csharp -e ‘new System.Net.WebClient ().DownloadString (“https://www.duplicati-monitoring.com”)’
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) —> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. —> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Certificate chain
 0 s:CN = duplicati-monitoring.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

References:

Transitioning to ISRG’s Root April 15, 2019
Transition to ISRG’s Root delayed until Jan 11 2021 June 11, 2020
Extending Android Device Compatibility for Let’s Encrypt Certificates December 21, 2020

I wonder if @crazy4chrissi has had a chance to look over my earlier post about changing chains?

1 Like

In my case that was definitely not the issue. I’m using Debian 11 with all updates.

Please blacklist expired DST Root CA X3 certificate is under discussion but might not actually get done.
If it got done, it would be similar to Ubuntu fix, which somehow made mono work even without cert-sync.
Mono Project could do a release for certificates, or use X509_V_FLAG_TRUSTED_FIRST, but will they?

https://ftp.openbsd.org/pub/OpenBSD/Changelogs/ChangeLog
looks like they’re setting above flag and also deleting expired DST Root CA X3 to solve various problems.

Point remains that Duplicati Monitoring could allow Duplicati Linux users without waiting for anybody else.

Looks like it’s been fixed in ca-certificates (20211004):

* Blacklist expired root certificate "DST Root CA X3" (closes: #995432)

Great. I hope that takes care of Debian. One thing I wonder about is whether simply updating the OS will update mono without a manual cert-sync. I was trying to look at my Ubuntu system, and ls -lu --full-time /usr/bin/cert-sync (time last read, implying a run) was 3 seconds after combined certificate file (location found by man update-ca-certificates. So maybe some automatic mechanism helped out…

$ ls -lu --full-time /usr/sbin/update-ca-certificates
-rwxr-xr-x 1 root root 5300 2021-10-03 21:34:24.314198859 -0400 /usr/sbin/update-ca-certificates
$ ls -l --full-time /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 199113 2021-10-03 21:36:15.826202323 -0400 /etc/ssl/certs/ca-certificates.crt
$ ls -lu --full-time /usr/bin/cert-sync
-rwxr-xr-x 1 root root 80 2021-10-03 21:36:18.474202406 -0400 /usr/bin/cert-sync

There’s a hook (at least on Debian) to automatically run mono cert-sync when ca-certificates is updated:

root@stimpy [/etc/ca-certificates/update.d]# ll
total 4
-rwxr-xr-x 1 root root 340 Feb 25  2020 mono-keystore*
root@stimpy [/etc/ca-certificates/update.d]# cat mono-keystore 
#!/bin/sh

set -e

[ -f /usr/bin/cert-sync ] || exit 0

echo "Updating Mono key store"

# always return true, even if the tool blows up for some reason.
# we don't want to mess with users at install-time. failure here
# isn't really fatal anyway, just inconvenient
/usr/bin/cert-sync /etc/ssl/certs/ca-certificates.crt || true

echo "Done"
$ ls -lu --full-time /etc/ca-certificates/update.d
total 8
-rwxr-xr-x 1 root root 2683 2021-10-03 21:36:15.866202325 -0400 jks-keystore
-rwxr-xr-x 1 root root  303 2021-10-03 21:36:18.466202405 -0400 mono-keystore

Ubuntu (not surprisingly) has it too, and mono-keystore was read 8 milliseconds before cert-sync.
The idea seems like a safer one than relying on mono updates, but I wonder if they ever disagree.

Thanks for pointing it out.

I think this might be from mono ca-certificates package, at least on Debian and derivatives:

Package: ca-certificates-mono (6.8.0.105+dfsg-3.2)

This package uses the hooks of the ca-certificates package to update the Mono keystore.

$ dpkg-query --search /etc/ca-certificates/update.d/mono-keystore
ca-certificates-mono: /etc/ca-certificates/update.d/mono-keystore
$ dpkg-query --listfiles ca-certificates-mono
/.
/etc
/etc/ca-certificates
/etc/ca-certificates/update.d
/etc/ca-certificates/update.d/mono-keystore
/usr
/usr/bin
/usr/bin/cert-sync
/usr/lib
/usr/lib/mono
/usr/lib/mono/4.5
/usr/lib/mono/4.5/cert-sync.exe
/usr/share
/usr/share/doc
/usr/share/doc/ca-certificates-mono
/usr/share/doc/ca-certificates-mono/changelog.Debian.gz
/usr/share/doc/ca-certificates-mono/copyright

Interesting description here thanks @ts678 !

I use Duplicati on Docker v2.0.6.3, any idea if there is a simple way to workaround?

I don’t run Docker. As a side note, Duplicati Monitoring is a completely separate project from Duplicati.
https://www.duplicati-monitoring.com/ says that, and gives some other contact paths, including email.

You have important data kept on your host (not in Docker, which is discarded during upgrades), right?
If so, you might see if there are enough tools in the Docker to try one of the documented workarounds.

The simplest workaround (which still violates the philosophy of don-t-change-the-Docker) would be for revised ca-certificates package to be released, then maybe it can be installed. Unfortunately, it’s stuck:

https://tracker.debian.org/pkg/ca-certificates

Things may change, but at the moment there’s a grave bug (seemingly in scripting, not in certificates):

Debian Bug report logs - #996005 – ca-certificates: fails upgrading when no new certs selected

Although Duplicati staff is busy (overly busy…) just with Duplicati, I think @drwtsn32 runs Duplicati by Duplicati Docker image (there are others – what’s yours?), so might be able to comment on your issue.

I would strongly prefer that Duplicati Monitoring serve a Linux Duplicati compatible certificate to its users.
This is the same request I make for any service that breaks Duplicati. Most storage has been OK so far.
Backblaze was covered earlier, and the big ones often don’t use Let’s Encrypt. Google is their own CA…

The solution for the official Duplicati Docker image is the same as posted above for Debian. You want to remove this file from within the container:

/usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
1 Like

Great its working now on my Docker instances! Thank you guys!

The only one still not working is the Duplicati plugin on my TrueNAS. Since this is FreeBSD based, the way certificates are managed slightly differs and I did not succeed.