There’s a hook (at least on Debian) to automatically run mono cert-sync when ca-certificates is updated:
root@stimpy [/etc/ca-certificates/update.d]# ll
total 4
-rwxr-xr-x 1 root root 340 Feb 25 2020 mono-keystore*
root@stimpy [/etc/ca-certificates/update.d]# cat mono-keystore
#!/bin/sh
set -e
[ -f /usr/bin/cert-sync ] || exit 0
echo "Updating Mono key store"
# always return true, even if the tool blows up for some reason.
# we don't want to mess with users at install-time. failure here
# isn't really fatal anyway, just inconvenient
/usr/bin/cert-sync /etc/ssl/certs/ca-certificates.crt || true
echo "Done"
Ubuntu (not surprisingly) has it too, and mono-keystore was read 8 milliseconds before cert-sync.
The idea seems like a safer one than relying on mono updates, but I wonder if they ever disagree.
I don’t run Docker. As a side note, Duplicati Monitoring is a completely separate project from Duplicati. https://www.duplicati-monitoring.com/ says that, and gives some other contact paths, including email.
You have important data kept on your host (not in Docker, which is discarded during upgrades), right?
If so, you might see if there are enough tools in the Docker to try one of the documented workarounds.
The simplest workaround (which still violates the philosophy of don-t-change-the-Docker) would be for revised ca-certificates package to be released, then maybe it can be installed. Unfortunately, it’s stuck:
Although Duplicati staff is busy (overly busy…) just with Duplicati, I think @drwtsn32 runs Duplicati by Duplicati Docker image (there are others – what’s yours?), so might be able to comment on your issue.
I would strongly prefer that Duplicati Monitoring serve a Linux Duplicati compatible certificate to its users.
This is the same request I make for any service that breaks Duplicati. Most storage has been OK so far.
Backblaze was covered earlier, and the big ones often don’t use Let’s Encrypt. Google is their own CA…
Great its working now on my Docker instances! Thank you guys!
The only one still not working is the Duplicati plugin on my TrueNAS. Since this is FreeBSD based, the way certificates are managed slightly differs and I did not succeed.
Hi, I also get a cert error when trying to run my desktop backups.
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) —> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. —> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
Thing is I am on Manjaro and since it’s not Debian based I can’t apply the above workarounds. I also couldn’t find anything tangible to do manually online, so I am at a bit of a loss here, and I don’t want to try taking risky initiatives since this is my daily driver machine.
Does anyone happen to know what I could do/read?
I have a fully updated system. Backups were fine yesterday.
Oh, I checked there but it wasn’t mentioned yet Thank you @ts678 for the help, it worked.
For anyone else interested, the commands for Arch mentioned in the link are those below, that should be done as root:
# blacklist the expired certificate
cp /etc/ssl/certs/DST_Root_CA_X3.pem /etc/ca-certificates/trust-source/blocklist/
# update the system wide trust store
update-ca-trust
# and finally sync the system trust store to mono
cert-sync /etc/ssl/certs/ca-certificates.crt
I’ve been having an issue for the last week, but I’m not sure if it is the same one from this thread or not.
Environment:
3 systems running Fedora 34, Duplicati - 2.0.6.100_canary_2021-08-11, mono 6.12.0-4
1 system running Windows 10, Duplicati - 2.0.6.100_canary_2021-08-11
All using backblaze for storage.
2 Fedora systems started failing Tuesday, November 30, the other Fedora system says it succeeded on the 30th, but then has failed since December 1. The Windows 10
system is not reporting any issues.
The error reported is:
System.AggregateException: One or more errors occurred. (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at /builddir/build/BUILD/mono-6.12.0.122/external/boringssl/ssl/handshake_client.c:1132 (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
[repeated multiples times - I can provide the full trace if desired]
I have attempted to blacklist the DST certificate - no change, so either it wasn’t the issue, or the attempt failed. Also, note that backups have been working through November so if the issue was the certificate expired in October, would this even be applicable?
I have also tried playing with forcing SSL3, TLS1.1, etc. (not saying I tried all of the them) - no help.
I tried “update-crypto-policies --set LEGACY” followed by a reboot - no help.
You could also ask what Backblaze support has to say. Above-named site (one of many you may trip on) has a Let’s Encrypt certificate with trouble-for-mono default chain, valid from 30 Nov 2021 02:22:14 UTC.
I did a “update-ca-trust extract”, extracted the “DST Root CA X3” section to a file in /etc/pki/ca-trust/source/blacklist, and then ran update-ca-trust to rebuild the cert db (or whatever it does). The output certainly appears to be honoring the blacklist - I get multiple lines saying “p11-kit: overriding trust for anchor in blacklist: DST.crt”. I also ran several cert-sync commands against files in the /etc/pki directory. To be extra safe, I then rebooted.
No web page data, no error. It just returns to the command prompt. Also, firefox has no trouble with the backblaze url. I will take a look at the mono-project link provided to see if it provides any further insight.
So you are saying that instead of a systemic issue, this would be specific to mono? If so I need to back-out the blacklist change. The Nov 30 date would be when 2 out of 3 systems started failing. Unless something was cached, I’m not sure why the 3rd took an extra 24 hours to fail.
As stated previously, the " csharp -e ‘new System.Net.WebClient ().DownloadString (“https://api.backblazeb2.com”)’" test results in no output and no error.
Let’s Encrypt Chain of Trust has that link described as the ISRG Root X1 self-signed version. Certificate Compatibility under Platforms that trust ISRG Root X1 doesn’t list Fedora, but might not be an exhaustive list. Regardless, thanks for finding and documenting the addition that solved the issue for you.
Thank you