Fixing a Lets Encrypt certificate error on RaspOS/arm64 for S3 storage

Hi
I ran into a problem when trying to use Duplicati - 2.0.6.3_beta on a Raspberry Pi 4 using the recent 64bit OS release. After many hours I was about to give up and beg for some help, but I finally solved it so I shall leave this here to help those who may come after me…

I could happily attach to my IDrive-e2 bucket using a Windows installation, but on the Pi I got the following error when I hit “Test connection”

Amazon.Runtime.AmazonServiceException: A WebException with status TrustFailure was thrown. ---> System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.Ausing uthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-6.12.0.182/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <33b19a7ad5234d94abf4fd9b47566616>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <33b19a7ad5234d94abf4fd9b47566616>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <33b19a7ad5234d94abf4fd9b47566616>:0 
  at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003b] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <79796cc3f14a4a2d81133207810e557f>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 
   --- End of inner exception stack trace ---
  at Amazon.Runtime.Internal.WebExceptionHandler.HandleException (Amazon.Runtime.IExecutionContext executionContext, System.Net.WebException exception) [0x0006e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.ExceptionHandler`1[T].Handle (Amazon.Runtime.IExecutionContext executionContext, System.Exception exception) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.ErrorHandler.ProcessException (Amazon.Runtime.IExecutionContext executionContext, System.Exception exception) [0x0005c] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00015] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.CallbackHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.Signer.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00006] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00030] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.RetryHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00040] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.CallbackHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.S3.Internal.AmazonS3KmsHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <79796cc3f14a4a2d81133207810e557f>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.EndpointResolver.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <79796cc3f14a4a2d81133207810e557f>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.Marshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00006] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <79796cc3f14a4a2d81133207810e557f>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.CallbackHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00007] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00012] in <79796cc3f14a4a2d81133207810e557f>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00012] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.MetricsHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0002b] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00006] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.Runtime.AmazonServiceClient.Invoke[TResponse] (Amazon.Runtime.AmazonWebServiceRequest request, Amazon.Runtime.Internal.InvokeOptionsBase options) [0x0007d] in <e28e89f25c1649a69062a2c53f89d718>:0 
  at Amazon.S3.AmazonS3Client.ListObjects (Amazon.S3.Model.ListObjectsRequest request) [0x0001c] in <79796cc3f14a4a2d81133207810e557f>:0 
  at Duplicati.Library.Backend.S3AwsClient+<ListBucket>d__12.MoveNext () [0x000a4] in <1150a6be8a7a4ff093c5f7e72bc45ad9>:0 
  at Duplicati.Library.Backend.S3+<List>d__29.MoveNext () [0x000d9] in <1150a6be8a7a4ff093c5f7e72bc45ad9>:0 
  at Duplicati.Library.Interface.BackendExtensions.TestList (Duplicati.Library.Interface.IBackend backend) [0x00017] in <fd3642a459884bd9a2412b4eda050109>:0 
  at Duplicati.Library.Backend.S3.Test () [0x00000] in <1150a6be8a7a4ff093c5f7e72bc45ad9>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <156011ea63b34859b4073abdbf0b1573>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00094] in <156011ea63b34859b4073abdbf0b1573>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <156011ea63b34859b4073abdbf0b1573>:0 

The IDrive certificate checked out OK in the browser but it was issued by Let’s Encrypt, which made me suspicious. I’d thought all the wrinkles with the expired intermediate cert last year had been fixed, but not it seems, when you’re using Mono!

Mono was at version 6.12.0.182 but it seemed to be implicated along with some other code it uses called “boringSSL”. Apparently, it can’t cope if a certificate trust route fails even when there’s another valid one available. Doh!

To cut to the finale, I had to work out how to remove all traces of the old, expired DST X3 certificate and then install the replacement Lets Encrypt (R3) one. I also installed their new ISRG Root X2 which is coming into service soon.

I tried so many ways to achieve this, I’m still not sure which commands actually did the job, but

sudo certmgr -list -c -m Trust | grep -A 3 -B 3 “SEARCH_TERM” helped see what was there

sudo rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt actually get rid of one version of the expired cert I had. I think it was the only one but it’s best to search around just in case.

sudo update-ca-certificates -f the -f means it removes any certificates no longer on the machine.

Best of luck.

2 Likes

thanks, these worked for me

sudo rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
sudo update-ca-certificates -f