I’m glad people are finding a workaround. To me it looks worse than the symptom; I will leave things alone until a proper update works its way through to me.
But my backups run, so I can afford to wait.
Enable X509_V_FLAG_TRUSTED_FIRST flag in BoringSSL #21233 is the latest-updated mono issue.
I’m hoping staying current with mono right from mono-project will resolve. Distros may do fixes slower.
No sucess on Synlogy yet. Hoever can across this blog which outlines and explains the issue
I guess you are running the native Duplicati package on Synology? I use the docker image on my Synology and the workaround worked for me.
My backups run - my only impact is missing duplicati-monitoring reports. I can leave it be now.
yeah for my sins its a DS218J no docker support for a small client
Sorry for joining the party so late. Yes, the root cause is that we use a letsencrypt certificate that is automatically updated regularly and their new root cert is not cross-signed anymore.
But: If this certificate causes errors on your client, you should really update your client. It is not like the “new” ISRG Root X1 certificate is something that popped up last month. Most browsers and operating systems trust it since years. If your client does not, it probably is badly outdated.
In our statistics we see that we continue to receive more and more backup reports. Currently about 16000 per day. So the vast majority of clients does not seem to have a problem with the new certificate.
1 Like
I do agree with you @Kahomono. No time to spend with workarounds either, and backup are working fine, of course without monitoring 
@Kahomono and @Rajstopy
The workaround is the fix. See my post with centos link
Updating the ca-certificates remove the offending expired certificate from your systems.
Using certmgr to recreate a copy of the new certificate’s in mono’s store. I found in some cases certmgr did not remove the expired cert and I had to manually delete same cert.
It will also prevent you from doing an upgrade to duplicati as the site use a Lets Encrypt Certificate
1 Like
Unfortunately, my mail server runs on Synology …
TL;DR I “think” this is a poor interaction between Let’s Encrypt default cert chain, OS certs, and mono.
This leaves multiple ways to fix, but the controllable one (though limited) is the certs that server sends.
I’m not a certificate wizard (but learning some), but the default one appears to be the cross signed one.
Wireshark shows that an Edge access to https://www.duplicati-monitoring.com gets the below:
Certificate: 3082056030820448a00302010202104001772137d4e942b8ee76aa3c640ab7300d06092a… (id-at-commonName=ISRG Root X1,id-at-organizationName=Internet Security Research Group,id-at-countryName=US)
signedCertificate
version: v3 (2)
serialNumber: 0x4001772137d4e942b8ee76aa3c640ab7
signature (sha256WithRSAEncryption)
Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
issuer: rdnSequence (0)
rdnSequence: 2 items (id-at-commonName=DST Root CA X3,id-at-organizationName=Digital Signature Trust Co.)
RDNSequence item: 1 item (id-at-organizationName=Digital Signature Trust Co.)
RDNSequence item: 1 item (id-at-commonName=DST Root CA X3)
validity
notBefore: utcTime (0)
notAfter: utcTime (0)
subject: rdnSequence (0)
subjectPublicKeyInfo
extensions: 7 items
algorithmIdentifier (sha256WithRSAEncryption)
Padding: 0
encrypted: 0a73006c966eff0e52d0aedd8ce75a06ad2fa8e38fbfc90a031550c2e56c42bb6f9bf4b4…
Note that the issuer is DST ROOT CA X3. The two versions of their certificate are described on page
Extending Android Device Compatibility for Let’s Encrypt Certificates showing default is cross-signed.
DST Root CA X3 Expiration (September 2021) talks about an Android exception. It doesn’t care about
DST Root CA X3 being expired. Other implementations do, such as older OpenSSL, and BoringSSL, which forked from it in 2014. mono uses BoringSSL. See the (oddly quiet) issue on it I mention earlier.
https://usage-reporter.duplicati.com/ suggests vast majority of Duplicati users use Windows not mono.
Resources for Certificate Chaining Help has a Workarounds for OpenSSL 1.0.2. link with three options.
1 is to remove the expired certificate as some are doing here, but I’m not sure how to keep it removed.
Ubuntu seems to have that problem solved by recently removing it from their ca-certificates packages.
2 is to ask the client to prefer the trust store certificates over the chain that the server sends. Without it, presumably the cross-sign of DST ROOT CA X3 is found and (except on Android) the access is failed.
The mono issue I cite asks them to do the change by using the X509_V_FLAG_TRUSTED_FIRST flag.
3 is for web server to get and use the alternative certificate chain that doesn’t go to DST ROOT CA X3.
It simply ends at ISRG Root X1 (self-signed) which presumably is trusted by most systems these days.
This breaks old Android, but that might not matter, and it probably solves the failures with current mono.
The limitation of this is that the server-side Let’s Encrypt workaround might need to be done elsewhere.
The procedure for asking for the alternative chain seemingly varies by ACME client (if it can even do it).
1 Like
For anyone running the native Duplicati package on Synology, I’ve documented how I solved it here (very easy, though it took huge amounts of flailing to get there). In brief:
sudo /volume1/@appstore/mono/bin/cert-sync \
/etc/ssl/certs/ca-certificates.crt
… and restart Duplicati. More details at the link.
1 Like
@conrad Thanks for that. I will try later on system in question. I tried to update the certs using mozroots and also removing the offending cert from crt file but maybe the new cert wasn’t in it.
I fixed the cert issue on another system by Upgrading to DSM 7 but it broke my Duplicati and I havent tried to fix it yet, so reluctant to do the same on the other system
Mozilla’s CA Certificate Program leads to
Mozilla Included CA Certificate List shows ISRG Root X1 has been included for about 5 years.
DST Root CA X3 is also there, even though it’s expired. Remember that Android doesn’t care.
To further make the case that Duplicati Monitoring should favor current mono over old Android:
FAQ: Security at Mono Project suggests a test like:
csharp -e ‘new System.Net.WebClient ().DownloadString (“https://www.nuget.org”)’
I’ll also quote from result of openssl s_client -connect <host>:443 to show the certificate chain:
PASS at https://api.backblazeb2.com which apparently sacrificed old Android, thankfully for us:
Certificate chain
0 s:CN = backblazeb2.com
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
PASS at https://valid-isrgrootx1.letsencrypt.org which is a test site for the new alternative chain:
Certificate chain
0 s:CN = valid-isrgrootx1.letsencrypt.org
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
FAIL at https://www.duplicati-monitoring.com presumably because default chain does old Android:
$ csharp -e ‘new System.Net.WebClient ().DownloadString (“https://www.duplicati-monitoring.com”)’
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) —> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. —> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
Certificate chain
0 s:CN = duplicati-monitoring.com
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
References:
Transitioning to ISRG’s Root April 15, 2019
Transition to ISRG’s Root delayed until Jan 11 2021 June 11, 2020
Extending Android Device Compatibility for Let’s Encrypt Certificates December 21, 2020
I wonder if @crazy4chrissi has had a chance to look over my earlier post about changing chains?
1 Like
In my case that was definitely not the issue. I’m using Debian 11 with all updates.
Please blacklist expired DST Root CA X3 certificate is under discussion but might not actually get done.
If it got done, it would be similar to Ubuntu fix, which somehow made mono work even without cert-sync.
Mono Project could do a release for certificates, or use X509_V_FLAG_TRUSTED_FIRST, but will they?
https://ftp.openbsd.org/pub/OpenBSD/Changelogs/ChangeLog
looks like they’re setting above flag and also deleting expired DST Root CA X3 to solve various problems.
Point remains that Duplicati Monitoring could allow Duplicati Linux users without waiting for anybody else.
Looks like it’s been fixed in ca-certificates (20211004):
* Blacklist expired root certificate "DST Root CA X3" (closes: #995432)
Great. I hope that takes care of Debian. One thing I wonder about is whether simply updating the OS will update mono without a manual cert-sync. I was trying to look at my Ubuntu system, and ls -lu --full-time /usr/bin/cert-sync (time last read, implying a run) was 3 seconds after combined certificate file (location found by man update-ca-certificates. So maybe some automatic mechanism helped out…
$ ls -lu --full-time /usr/sbin/update-ca-certificates
-rwxr-xr-x 1 root root 5300 2021-10-03 21:34:24.314198859 -0400 /usr/sbin/update-ca-certificates
$ ls -l --full-time /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 199113 2021-10-03 21:36:15.826202323 -0400 /etc/ssl/certs/ca-certificates.crt
$ ls -lu --full-time /usr/bin/cert-sync
-rwxr-xr-x 1 root root 80 2021-10-03 21:36:18.474202406 -0400 /usr/bin/cert-syncThere’s a hook (at least on Debian) to automatically run mono cert-sync when ca-certificates is updated:
root@stimpy [/etc/ca-certificates/update.d]# ll
total 4
-rwxr-xr-x 1 root root 340 Feb 25 2020 mono-keystore*
root@stimpy [/etc/ca-certificates/update.d]# cat mono-keystore
#!/bin/sh
set -e
[ -f /usr/bin/cert-sync ] || exit 0
echo "Updating Mono key store"
# always return true, even if the tool blows up for some reason.
# we don't want to mess with users at install-time. failure here
# isn't really fatal anyway, just inconvenient
/usr/bin/cert-sync /etc/ssl/certs/ca-certificates.crt || true
echo "Done"