Using MinIO (S3) backend with Letsencrypt (wildcard) certificate

Support
1

I have been running Duplicati against a MinIO backend for years now. I recently changed the certificate of the MinIO backend from a bespoke to a wildcard cert, but I forgot that in the past I had to tell all Duplicatis to either ignore cert errors or tell it to accept a certain signature. So, backups started failing until I realised what was going on.

Is it somehow fixable that Duplicati ‘just works’ when the S3 backend has a proper (Letsencrypt) certificate without having to tell it on each Duplicati instance to ignore cert errors (which isn’t the best security practice after all)?

2

Hello

I don’t quite understand how Duplicati could fail with a ‘true’ certificate if you had configured it before to accept any certificate. On the other hand if you had instructed it to bind to a given hash, it seems reasonable to expect that the software will respect that and do what you asked for, that is, reject all other certificates even if they are legitimate Internet approved certificates.
So yes, Duplicati should work without any extra options with a LetsEncrypt certificate.

3

When did that start? On September 30 2021, DST Root CA X3 certificate expired, so broke mono.
Duplicati doesn’t handle the certificates itself. On Windows, Windows does. On others, mono does.
If on mono with csharp installed, you can maybe test per their FAQ: Security to see how TLS does.
Working around problems with Lets Encrypt certificates is distro-specific. You can search for yours.

Or maybe it’s something else.

Got any specific error information from Duplicati? That would also help identify what’s going wrong.

4

It started when I replaced the certificate from a specific foo.rna.nl by a wild card domain one: *.rna.nl. I can reliably trigger this behaviour by switching certs. I have now set the backup to accept-any-ssl-certificate but that doesn’t work, either. So I have reinstalled the specific certificate for now.

Amazon.Runtime.AmazonServiceException: A WebException with status SecureChannelFailure was thrown. ---> System.Net.WebException: Error: SecureChannelFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Security.Interface.TlsException: Unknown Secure Transport error `PeerHandshakeFail'.
  at Mono.AppleTls.AppleTlsContext.CheckStatusAndThrow (Mono.AppleTls.SslStatus status, Mono.AppleTls.SslStatus[] acceptable) [0x000dc] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at Mono.AppleTls.AppleTlsContext.ProcessHandshake () [0x0002e] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0 
  at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003b] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <79796cc3f14a4a2d81133207810e557f>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.BackendManager.List () [0x00049] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x0000d] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00000] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0011d] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01048] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00074] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00349] in <156011ea63b34859b4073abdbf0b1573>:0

Note, I am running Duplicati - 2.0.6.3_beta_2021-06-17 as that one was stable for me (so far…)

5

@gctwnl

you said that you are running Duplicati ‘for years’. You upgraded Duplicati all right, but did you upgrade Mono too ? You should run 6.12 at least.

6

The setup over time is very confusing. Before going to a wildcard cert, you had to use a workaround.
That was “in the past”. Any chance you have a vague idea of when the non-wildcard-cert issue was?

and now that you’re back on a specific certificate, is workaround still needed? What’s error without it?
Expectation is that it will fail due to the certificate expiration mentioned. Please run the test described.
Original post sounds like you had an old unsolved issue, yet focus on new issue. Old one may matter.

Do your S3 hosts simply replace * or use multiple levels? I think a wildcard only allows one more level.
Wildcard SSL on sub-subdomain [closed]

7

Mono version:

gerben@hermione% port installed|grep mono
  mono @6.12.0.182_0 (active)
8

I have now removed all the ssl-flags. No difference. I am connecting to foo.rna.nl and with a specific cert for foo.rna.nl the connection works and with a wildcard one for *.rna.nl it fails. Note, the same cert works fine in all other settings.

9

Maybe good to add for completeness, though I think it is not relevant as this should work as well: I am using nginx to do the SSL-termination. nginx then passes on the unencrypted traffic to MinIO via localhost.

10

It didn’t sometime past, per original post? Any idea why? Was something mis-said or misread?
This is rather critical to know. Specific cert used to need options (yes/no?) but now it does not?

I’m asking again for you to test it in a mono setting, per the FAQ from mono project (see above).
A failure seems less certain now that I’ve attempted to read mono source code (not my area…),
however there are plenty of macOS users visible on my Internet search hitting that expired cert,
which is on your chain, assuming your wildcard cert is also on https://www.rna.nl. Result of
Qualys SSL Labs test of that URL shows certificates and certificate paths, some to expired cert.
Ignore their claim that says it’s harmless. It can break mono. Start at issue I linked, follow to next.
Your public site also needs TLS 1.3, which mono can’t do. Maybe your internal sites do TLS 1.2?

The best vague clue I can see in your stack trace (thanks for that) was the PeerHandshakeFail.

which sounds disturbingly like it might be necessary to go in for a deep look using some of these

Wireshark or its commandline version tshark or maybe do capture with an old tool like tcpdump.

In theory you could post a capture of the encrypted session and not reveal anything not visible if over public Internet connection and TLS, but I think the SNI is clear text and some addresses can be seen.

11

It didn’t sometime past, per original post? Any idea why? Was something mis-said or misread?
This is rather critical to know. Specific cert used to need options (yes/no?) but now it does not?

In the dim past, I used a self-signed cert for which I needed an SSL flag. I replaced the self-signed with a bespoke LE cert years ago and never removed the (now unnecessary) SSL flag. I now replaced the bespoke cert by a wildcard cert.

12

mono is ok on the system (tested according to the FAQ):

gerben@hermione% csharp -e 'new System.Net.WebClient ().DownloadString ("https://www.nuget.org")'
"<!DOCTYPE html>\r
<html lang="en">\r
<head>\r
    <meta charset="utf-8" />\r
...
13

The boringssl library version used by the Minio version linked by Duplicati does not support ECC keys.

14

You’d use your own actual URL for the test. Here’s an example of one where a failure occurs:

$ csharp -e ‘new System.Net.WebClient ().DownloadString (“https://api.backblazeb2.com”)’
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) —> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. —> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at /build/mono-Du4GKE/mono-6.8.0.105+dfsg/external/boringssl/ssl/handshake_client.c:1132
at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00064] in :0

15

Is that relevant to below? I’m still trying to sort out what’s what…

16

Progress:

Actually I did this, but incorrectly and then posted the example here. https://www.rna.nl/ on the outside gets NAT forwarded to https://foo.rna.nl:8445/ inside and I forgot the internal port. When I did, I could trigger the error with mono on the wildcard cert:

System.Net.WebException: Error: SecureChannelFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Security.Interface.TlsException: Unknown Secure Transport error `PeerHandshakeFail'.
  at Mono.AppleTls.AppleTlsContext.CheckStatusAndThrow (Mono.AppleTls.SslStatus status, Mono.AppleTls.SslStatus[] acceptable) [0x000dc] in <31ab9cbd49d84bc9a3c66a80af36e0cb>:0

Maybe gpatel-fr is correct?

The boringssl library version used by the Minio version linked by Duplicati does not support ECC keys.

The fact that nginx does the termination is not relevant (it does it both for the bespoke as the wildcard cert)

17

I think so. Using a proxy to terminate the connection is not relevant.
cat http.cs

using System;
using System.Net.Http;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        Console.WriteLine(args[0]);
        using (var client = new HttpClient())
        {
           var result = await client.GetAsync(args[0]);
           Console.WriteLine(result.StatusCode);
           Console.WriteLine(result.Content.Headers);
        }
    }
}

mono bin/Debug/http.exe https://duplicati.com

https://duplicati.com
OK
Content-Type: text/html; charset=utf-8
Content-Length: 11445
Last-Modified: Mon, 16 Jan 2023 21:17:16 GMT
Expires: Thu, 26 Jan 2023 17:29:35 GMT

mono bin/Debug/http.exe https://www.rna.nl

https://www.rna.nl

Unhandled Exception:
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
  at /build/mono-6.12.0.182/external/boringssl/ssl/tls_record.c:462

openssl s_client -connect duplicati.com:443 | grep "Peer signa"
(...)
Peer signature type: RSA-PSS

openssl s_client -connect www.rna.nl:443 | grep "Peer signa"
(...)
Peer signature type: ECDSA
18

This is the correct answer/solution. When I force certbot to create an old fashioned RSA cert

certbot renew \              
    --key-type rsa \
    --keep-until-expiring --non-interactive --expand \
    --server https://acme-v02.api.letsencrypt.org/directory \
    --agree-tos \
    --cert-name rna.nl --force-renewal

the problem goes away.

19

That’s the end goal, so it’s wonderful to hear. Did move to ECC just happen to coincide with wildcard?

20

Yes. And I can’t just post “Yes” because the forum requires 20 characters minimum…

1 Like