Asustor NAS - Certificate TrustFailure

[noldomino]

Hello,
For several days I’m trying to resolve well-known TrustFailure issue related to “DST_Root_CA_X3” certificate. My backup setup utilize Duplicati & Jottacloud on Asustor AS6460T NAS. Please help. I’ve exhausted all possibilities I’ve found.
Error:

System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /asustor/trunk_2021_01_12/x64_g3/x64_g3/objs/mono-6.12.0.107/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <652d36f38550428c85c9b233c47d70a6>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <652d36f38550428c85c9b233c47d70a6>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <652d36f38550428c85c9b233c47d70a6>:0 
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <652d36f38550428c85c9b233c47d70a6>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <652d36f38550428c85c9b233c47d70a6>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <652d36f38550428c85c9b233c47d70a6>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00088] in <652d36f38550428c85c9b233c47d70a6>:0 
  at System.Net.WebClient.DownloadFile (System.String address, System.String fileName) [0x00008] in <652d36f38550428c85c9b233c47d70a6>:0 
  at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadFile(string,string)
  at Duplicati.Library.AutoUpdater.UpdaterManager.CheckForUpdate (Duplicati.Library.AutoUpdater.ReleaseType channel) [0x000ee] in <8d4cb1693e00483189d3952c3f0ed20f>:0

With the error in place I can’t update Duplicati to new canary release thus I can’t backup to Jottacloud. I’ve tried every solution I’ve found on the internet but nothing worked permanently or I do it improperly. My linux knowledge is very limited.
Main solutions tried:

1. Deleting “DST ROOT CA X3” cert content from ca-certificates.crt as per Error in updater - Trust Failure - SSL error - #5 by seanws78

Open the file ca-certificates.crt
/etc/ssl/certs # vi ca-certificates.crt
sudo cert-sync /etc/ssl/certs/ca-certificates.crt

It is not working at all.

2. As above but with user store

sudo cert-sync --user /etc/ssl/certs/ca-certificates.crt

Not working either.

3. Banning the “DST ROOT CA X3” cert and updating trust as in:
   Error in updater - Trust Failure - SSL error - #3 by obones

sudo trust extract --format=pem-bundle --filter='pkcs11:id=%C4%A7%B1%A4%7B%2C%71%FA%DB%E1%4B%90%75%FF%C4%15%60%85%89%10;type=cert' /etc/ca-certificates/trust-source/blocklist/untrusted_authority.pem
sudo update-ca-trust
sudo cert-sync /etc/ssl/certs/ca-certificates.crt

Not possible. Incompatible with Asustor.

4. Deleting cert from Trust strore with hash
   Error in updater - Trust Failure - SSL error - #10 by HybridAU

sudo certmgr -del -c -v -m Trust 5BCAA1C2780F0BCB5A90770451D96F38963F012D

It removes the certificate but gives no effect. Certificate is back on the next cert-sync.

5. Editing config file

sudo dpkg-reconfigure ca-certificates
edit the file /etc/ca-certificates.conf. 
sudo update-ca-certificates

Not possible on Asustor (no config file; update-ca-certificates not working)

6. Duplicati advanced options “accept-any-ssl-certificate” and “accept-specified-ssl-hash” doesn’t work

7. Deleting certificate as per:
   Failed to connect: Error: TrustFailure (Authentication failed, see inner exception.) - #6 by drwtsn32

Delete file:
/usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
And run:
update-ca-certificates

Can’t run “update-ca-certificates” on Asustor thus cert-sync was used:

sudo cert-sync /etc/ssl/certs/ca-certificates.crt

Actually it worked and Duplicati update to last canary was successful. I changed update channel back to Beta. Backups were running properly for 2 days. Unfortunately after NAS reboot issue was back again with Duplicati downgraded to beta (without prompt).
Using the solution for the second time gives nothing.

I’ve spent many hours on the issue so far without results.
I’ll appreciate any help to resolve it finally.
Thank you in advance.

Unix 5.4.0.0 (Asustor)
Duplicati 2.0.6.3
Mono 6.12.0.107

[noldomino]

After some more tryouts and reboot I currently have following error after cert-sync:

cert-sync /etc/ssl/certs/ca-certificates.crt
Mono Certificate Store Sync - version 6.12.0.107
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 126, your new list has 127
Warning: Could not import O=Digital Signature Trust Co., CN=DST Root CA X3
System.UnauthorizedAccessException: Access to the path "/usr/share/.mono/certs/Trust/ski-C4A7B1A47B2C71FADBE14B9075FFC415608589106B40F82E86393089BA27A3D680B0AF44.cer" is denied.
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <0c9f397dc91544218ef5fe2a2fa0ea36>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize) [0x00000] in <0c9f397dc91544218ef5fe2a2fa0ea36>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int)
  at System.IO.File.Create (System.String path, System.Int32 bufferSize) [0x00000] in <0c9f397dc91544218ef5fe2a2fa0ea36>:0
  at System.IO.File.Create (System.String path) [0x00000] in <0c9f397dc91544218ef5fe2a2fa0ea36>:0
  at Mono.Security.X509.X509Store.Import (Mono.Security.X509.X509Certificate certificate) [0x00058] in <06c31303b9934a90ad7e4d2ed0f692dd>:0
  at Mono.Tools.CertSync.ImportToStore (Mono.Security.X509.X509CertificateCollection roots, Mono.Security.X509.X509Store store) [0x00050] in <7ed2d3eb07c04e469e23c4d129804742>:0
Import process completed.

Importing into BTLS system store:
I already trust 127, your new list has 127
Import process completed.

Mentioned file was not deleted manually.

[seanws78]

I have a fix but you have to run it after every reboot. I don’t know what else to do.

My solution is here

Error in updater - Trust Failure - SSL error

Hi. Thanks so much for this help. I managed to delete the offending certificate with help from this link https://github.com/duplicati/duplicati/issues/4650 On the ASUSTOR NAS using SSH Open the file ca-certificates.crt /etc/ssl/certs # vi ca-certificates.crt The problem was that it does not give you the certificate name before the BEGIN CERTIFICATE so i had to search for the actual certificate using parts of the actual certificate and checking that it matched with the one in the post. I …

[seanws78]

The start of the certificate is shown below. If you just search for the MIIDS there is only one

------BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/

[noldomino]

@seanws78 Indeed the cert content was there.
I tried it several times before. In my case ca-certificates.crt wasn’t repopulated with this certificate for a long time (even after reboots), thus I stopped checking. I have no idea what changed in the meantime. I’ll observe if the soulution is permanent for me this time.
Now I managed to update duplicati.
Thank you very much