I’m using the duplicatie Docker image, and since a few days, all my backups towards Backblaze started failing with a TrustFailure error. After taking some packet captures en looking at the SSL traffic, I see that the certificate for api.backblazeb2.com was recently renewed (30/11/2021).
See SSL Server Test: api.backblazeb2.com (Powered by Qualys SSL Labs)
Is there a way to:
A) See the list of trusted certificates used Duplicati?
B) Troubleshoot why it’s not trusting the certificate?
I did some more tests (all performed inside the container):
csharp -e 'new System.Net.WebClient ().DownloadString ("https://api.backblazeb2.com")' fails.cert-sync /etc/ssl/certs/ca-certificates.crt followed by the previous command, still fails.I came to ask the same thing. I updated mono but it didn’t help. If it’s useful, I’m getting the following log:
Failed: Error: TrustFailure (Authentication failed, see inner exception.)
Details: System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at /build/mono-6.12.0.122/external/boringssl/ssl/handshake_client.c:1132
at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
--- End of inner exception stack trace ---
at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
--- End of inner exception stack trace ---
at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebOperation.Run () [0x0009a] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Duplicati.Library.Main.BackendManager.List () [0x00049] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x0000d] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00000] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0011d] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01048] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011c] in <e60bc008dd1b454d861cfacbdd3760b9>:0
Log data:
2021-12-04 10:35:10 +00 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at /build/mono-6.12.0.122/external/boringssl/ssl/handshake_client.c:1132
at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
--- End of inner exception stack trace ---
at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
--- End of inner exception stack trace ---
at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebOperation.Run () [0x0009a] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <9c6e2cb7ddd8473fa420642ddcf7ce48>:0
at Duplicati.Library.Main.BackendManager.List () [0x00049] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x0000d] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00000] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0011d] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00452] in <e60bc008dd1b454d861cfacbdd3760b9>:0
I also came on here to ask the same question. Interestingly though only one of my servers is exhibiting this behaviour All the others that are the same config / mono version etc and that also backup to Backblaze are working fine. The server in question also has a onedrive backup and that is working fine too,
It’s been the last 3 nights that have failed so ties in with this renewed ssl cert that @jobec has identified.
Seems like Backblaze is using Let’sEncrypt certificates.
In the certificate chain they are sending is a certificate (ISRG Root X1) that leads up to an expired root CA (DST Root CA X3).
Some time ago, that intermediate CA “ISRG Root X1” has been re-issues as a root CA and added to all browsers. (DST Root CA X3 Expiration (September 2021) - Let's Encrypt) And those browsers (and recent openssl based stuff) ignore the certificate in the chain sent by the server and use the first matching certificate in their trust store instead of walking the server-sent chain until the end.
It seems like Mono, the thing Duplicati is build on, is using BoringSSL, which does not do that and fails validation. (See here: Enable X509_V_FLAG_TRUSTED_FIRST flag in BoringSSL · Issue #21233 · mono/mono · GitHub). So, no matter if you remove the expired root CA from the mono trust store, and add the new “ISRG Root X1” root CA, it will not work.
I did some test to a couple of other Let’s Encrypt based sites, and they al give the same issue. So it seems related maybe to the “certbot” stuff from Let’s Encrypt still sending that faulty intermediate CA certificate.
Meanwhile I also opened a ticket with Backblaze, hoping they can fix it from their side somehow.
Only workaround for now, it seems, is enabling the accept-any-ssl-certificate option in duplicati.
I wonder why this isn’t impacting my other Backblaze backups?
Thanks for the tip about setting the accept-any-ssl-certificate. I have managed to get a backup done now.
Are you making those other backups with Duplicati also?
Sounds like the same issue that was reported here:
The workaround is to delete a certain certificate.
I had this problem on Fedora 35 but not on Ubuntu 21.10. It seems fedora doesn’t keep the mono trust store properly updated.
Removing the offending certificate fixes the problem: see the thread linked by @drwtsn32 above.
Ok… finally managed to fix it for my container.
It seems there was a .mono folder with certificates on the /data volume mounted in the container.
I removed that folder and then ran the commands referenced in the post of @drwtsn32 and now it seems to work. I guess there was some left over stuff from trying to get it to work. Didn’t anticipate it being in the data volume.
root@server:/root# docker exec -ti -u root duplicati bash
root@1ac12dc7cd9b:/# csharp -e 'new System.Net.WebClient ().DownloadString ("https://api.backblazeb2.com")'
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at /build/mono-6.12.0.107/external/boringssl/ssl/handshake_client.c:1132
...
root@1ac12dc7cd9b:/# apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 Packages [309 kB]
Get:4 https://download.mono-project.com/repo/debian stable-buster/snapshots/6.12.0.107 InRelease [5888 B]
Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7906 kB]
Get:7 https://download.mono-project.com/repo/debian stable-buster/snapshots/6.12.0.107/main amd64 Packages [48.8 kB]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8524 kB in 3s (3297 kB/s)
Reading package lists... Done
W: Conflicting distribution: https://download.mono-project.com/repo/debian stable-buster/snapshots/6.12.0.107 InRelease (expected stable-buster/snapshots/6.12.0.107 but got buster)
root@1ac12dc7cd9b:/# apt-get install ca-certificates
Reading package lists... Done
Building dependency tree
Reading state information... Done
ca-certificates is already the newest version (20200601~deb10u2).
ca-certificates set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
root@1ac12dc7cd9b:/# rm -f /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
root@1ac12dc7cd9b:/# update-ca-certificates
Updating certificates in /etc/ssl/certs...
W: /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt not found, but listed in /etc/ca-certificates.conf.
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Updating Mono key store
Mono Certificate Store Sync - version 6.12.0.107
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
Importing into legacy system store:
I already trust 137, your new list has 136
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Importing into BTLS system store:
I already trust 137, your new list has 136
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
root@1ac12dc7cd9b:/$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://api.backblazeb2.com/")'
"<!DOCTYPE html>
<html lang="en">
<head>
...
I see you have the fix now, great. But yes, I have 6 identical servers all with the same version of mono and the same version of duplicati all backing up to backblaze and only one has this problem.
I will go fix it now. 
I got hit with this too. I used the “accept any certificate option” and now the connection is working.
Any idea how much of a drop in security this option is?
I’ve had a couple of years of flawless backups between OMV / Duplicati / and B2. I’m surprised at this interruption, but many gotchas in many cloud services always seem to point back to a simple certificate change.
Well, it sort of takes away the security TLS/SSL/HTTPS gives you.
Accepting any certificate means you’re not verifying anymore who you’re talking with. If someone manages to get in between your duplicati instance and it’s backup target, he can play man in the middle and record/manipulate your backup or restore.
How much more insecure it actually is, depends on what’s all in between. It’s good for testing or personal/home things, but I would disable it again if backups are critical to you.
Try removing that obsolete and expired certificate from mono’s trust store, before disabling certificate verification all together.
That may be a little advanced for me - but I’ll give it a shot.
I see what you posted above and will look into that. Wondering if there’s a GUI way to do this for non-commandprompt people such as myself.
In the Duplicati web interface, go to Settings and scroll down to Default Options, then select Add Advanced Option. I selected accept-any-ssl-certificate as recommended above. My backups are now working again.
Hope this works for you too.
Do you have any more specific instructions? There’s A LOT non-Fedora specific going on in @drwtsn32’s linked thread.
EDIT: Nevermind eventually figured it out Duplicati B2 Backend Broken - Fedora 35 · Issue #4650 · duplicati/duplicati · GitHub
Is this fix permanent? Or do we need to do it each time we update the Docker image?
Is this going to be a fix in future Duplicati images?