Http send report errors / duplicati-monitoring

I don’t really understand how this works because the DST_Root_CA_X3 isn’t anywhere in the cert chain for the duplicati-monitoring site. But it does work! Thanks. I spent a few hours yesterday trying to fix this.

Glad it helped! I honestly don’t fully understand it either, but then again, I am in no way an encryption expert.

I just followed the notes provided by @ccmgr but translated it to something that worked on Debian 10. Thank them for the initial idea!

Yes that’s correct on a web browser but if you run monos certmgr -ssl -c https://duplicati-monitoring.com it will show DST_Root_CA_X3 in the chain.

I applied the same fix to a Centos 7 System but cert-sync did not totally remove the offending certs.
certmgr report the DST cert not installed but on future investigation physical cert files where in the store…
Had to take a brute force option
Mono certs stores are in .~/.config/.mono/ or /usr/share/.mono depending on setup
The folders may have certs in either or in my case both certs\Trust and new-certs\Trust. I had to grep to find the DST cert files and move/remove them.

You dont have to run a backup to test as checking for an update will throw the same error

2 Likes

Interesting… wonder why there is a discrepancy…

I’m glad people are finding a workaround. To me it looks worse than the symptom; I will leave things alone until a proper update works its way through to me.

But my backups run, so I can afford to wait.

Enable X509_V_FLAG_TRUSTED_FIRST flag in BoringSSL #21233 is the latest-updated mono issue.
I’m hoping staying current with mono right from mono-project will resolve. Distros may do fixes slower.

No sucess on Synlogy yet. Hoever can across this blog which outlines and explains the issue

I guess you are running the native Duplicati package on Synology? I use the docker image on my Synology and the workaround worked for me.

My backups run - my only impact is missing duplicati-monitoring reports. I can leave it be now.

yeah for my sins its a DS218J no docker support for a small client

I had the same issue, and the solution from @hairlesshobo worked for me.

Sorry for joining the party so late. Yes, the root cause is that we use a letsencrypt certificate that is automatically updated regularly and their new root cert is not cross-signed anymore.

But: If this certificate causes errors on your client, you should really update your client. It is not like the “new” ISRG Root X1 certificate is something that popped up last month. Most browsers and operating systems trust it since years. If your client does not, it probably is badly outdated.

In our statistics we see that we continue to receive more and more backup reports. Currently about 16000 per day. So the vast majority of clients does not seem to have a problem with the new certificate.

1 Like

I do agree with you @Kahomono. No time to spend with workarounds either, and backup are working fine, of course without monitoring :unamused:

@Kahomono and @Rajstopy
The workaround is the fix. See my post with centos link
Updating the ca-certificates remove the offending expired certificate from your systems.
Using certmgr to recreate a copy of the new certificate’s in mono’s store. I found in some cases certmgr did not remove the expired cert and I had to manually delete same cert.
It will also prevent you from doing an upgrade to duplicati as the site use a Lets Encrypt Certificate

1 Like

Unfortunately, my mail server runs on Synology …

TL;DR I “think” this is a poor interaction between Let’s Encrypt default cert chain, OS certs, and mono.
This leaves multiple ways to fix, but the controllable one (though limited) is the certs that server sends.

I’m not a certificate wizard (but learning some), but the default one appears to be the cross signed one.
Wireshark shows that an Edge access to https://www.duplicati-monitoring.com gets the below:

Certificate: 3082056030820448a00302010202104001772137d4e942b8ee76aa3c640ab7300d06092a… (id-at-commonName=ISRG Root X1,id-at-organizationName=Internet Security Research Group,id-at-countryName=US)
    signedCertificate
        version: v3 (2)
        serialNumber: 0x4001772137d4e942b8ee76aa3c640ab7
        signature (sha256WithRSAEncryption)
            Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
        issuer: rdnSequence (0)
            rdnSequence: 2 items (id-at-commonName=DST Root CA X3,id-at-organizationName=Digital Signature Trust Co.)
                RDNSequence item: 1 item (id-at-organizationName=Digital Signature Trust Co.)
                RDNSequence item: 1 item (id-at-commonName=DST Root CA X3)
        validity
            notBefore: utcTime (0)
            notAfter: utcTime (0)
        subject: rdnSequence (0)
        subjectPublicKeyInfo
        extensions: 7 items
    algorithmIdentifier (sha256WithRSAEncryption)
    Padding: 0
    encrypted: 0a73006c966eff0e52d0aedd8ce75a06ad2fa8e38fbfc90a031550c2e56c42bb6f9bf4b4…

Note that the issuer is DST ROOT CA X3. The two versions of their certificate are described on page
Extending Android Device Compatibility for Let’s Encrypt Certificates showing default is cross-signed.
DST Root CA X3 Expiration (September 2021) talks about an Android exception. It doesn’t care about
DST Root CA X3 being expired. Other implementations do, such as older OpenSSL, and BoringSSL, which forked from it in 2014. mono uses BoringSSL. See the (oddly quiet) issue on it I mention earlier.

https://usage-reporter.duplicati.com/ suggests vast majority of Duplicati users use Windows not mono.

Resources for Certificate Chaining Help has a Workarounds for OpenSSL 1.0.2. link with three options.

1 is to remove the expired certificate as some are doing here, but I’m not sure how to keep it removed.
Ubuntu seems to have that problem solved by recently removing it from their ca-certificates packages.

2 is to ask the client to prefer the trust store certificates over the chain that the server sends. Without it, presumably the cross-sign of DST ROOT CA X3 is found and (except on Android) the access is failed.
The mono issue I cite asks them to do the change by using the X509_V_FLAG_TRUSTED_FIRST flag.

3 is for web server to get and use the alternative certificate chain that doesn’t go to DST ROOT CA X3.
It simply ends at ISRG Root X1 (self-signed) which presumably is trusted by most systems these days.

This breaks old Android, but that might not matter, and it probably solves the failures with current mono.
The limitation of this is that the server-side Let’s Encrypt workaround might need to be done elsewhere.
The procedure for asking for the alternative chain seemingly varies by ACME client (if it can even do it).

1 Like

For anyone running the native Duplicati package on Synology, I’ve documented how I solved it here (very easy, though it took huge amounts of flailing to get there). In brief:

sudo /volume1/@appstore/mono/bin/cert-sync \
    /etc/ssl/certs/ca-certificates.crt

… and restart Duplicati. More details at the link.

1 Like

@conrad Thanks for that. I will try later on system in question. I tried to update the certs using mozroots and also removing the offending cert from crt file but maybe the new cert wasn’t in it.

I fixed the cert issue on another system by Upgrading to DSM 7 but it broke my Duplicati and I havent tried to fix it yet, so reluctant to do the same on the other system

Mozilla’s CA Certificate Program leads to
Mozilla Included CA Certificate List shows ISRG Root X1 has been included for about 5 years.
DST Root CA X3 is also there, even though it’s expired. Remember that Android doesn’t care.

To further make the case that Duplicati Monitoring should favor current mono over old Android:

FAQ: Security at Mono Project suggests a test like:

csharp -e ‘new System.Net.WebClient ().DownloadString (“https://www.nuget.org”)’

I’ll also quote from result of openssl s_client -connect <host>:443 to show the certificate chain:

PASS at https://api.backblazeb2.com which apparently sacrificed old Android, thankfully for us:

Certificate chain
 0 s:CN = backblazeb2.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

PASS at https://valid-isrgrootx1.letsencrypt.org which is a test site for the new alternative chain:

Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

FAIL at https://www.duplicati-monitoring.com presumably because default chain does old Android:

$ csharp -e ‘new System.Net.WebClient ().DownloadString (“https://www.duplicati-monitoring.com”)’
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) —> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. —> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Certificate chain
 0 s:CN = duplicati-monitoring.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

References:

Transitioning to ISRG’s Root April 15, 2019
Transition to ISRG’s Root delayed until Jan 11 2021 June 11, 2020
Extending Android Device Compatibility for Let’s Encrypt Certificates December 21, 2020

I wonder if @crazy4chrissi has had a chance to look over my earlier post about changing chains?

1 Like

In my case that was definitely not the issue. I’m using Debian 11 with all updates.