Windows service + minio with ssl error

davegold · September 9, 2017, 2:10am

I set up a new Minio server with SSL keys, pointed Duplicati to it, backed up and restored no problem.

Converted Duplicati to run as a service, and am now seeing failures connecting to Minio.

Amazon.Runtime.AmazonServiceException: A WebException with status SecureChannelFailure was thrown. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.GetResponse()
   at Amazon.Runtime.Internal.HttpRequest.GetResponse()
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext executionContext, WebException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3KmsHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointResolver.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Marshaller.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller)
   at Amazon.S3.AmazonS3Client.ListObjects(ListObjectsRequest request)
   at Duplicati.Library.Backend.S3Wrapper.ListBucket(String bucketName, String prefix)
   at Duplicati.Library.Backend.S3.List()
   at Duplicati.Library.Main.BackendManager.DoList(FileEntryItem item)
   at Duplicati.Library.Main.BackendManager.ThreadRun()```

Any thoughts?

--Dave

dgcom · September 9, 2017, 5:06am

What SSL certificate did you use?
Self-signed or from some trusted CA?
The difference is because Windows have several certificate stores where it keeps root and intermediate certificates to create trust chain and verify - there is a per-user store, machine store and service store.
When you first set things up, Duplicati run in the context of your user and was able to verify server certificate.
When you converted to the service and use LocalSystem account, it most probably uses machine account (or service account).
You have two options:

quick one - change service account to yout user (which was used for testing)
get signing certificates (or server certificate if self-signed) and place them in appropriate places in machine certificate store

davegold · September 9, 2017, 4:50pm

SSL certificate is one from a trusted CA (namecheap.com) that I’m currently using on a web server I host.
I can hit the web page using https with no errors.

I had already tried using my username (a local administrator), and I’ve just reproduced it using my desktop which is physically on the same LAN at the S3 target (minio)

When I run the test from Duplicati, the S3 server (minio) shows:
ERRO[0749] TLS handshake failed with new connection 192.168.2.162:55696 at server 192.168.2.216:50002 cause=tls: client offered an unsupported, maximum protocol version of 301 source=[listener.go:172:github.com/minio/minio/pkg/http.(*httpListener).start.func2()]

per TLS handshake error, client offered an unsupported, maximum protocol version of 301 - Help - Caddy Community, this is TLS 1.0. So I tried to set the variable in Duplicati to use TLS 1.2 only, and got the same result.

I’m thinking that I need to either put Duplicati in debug mode, or run tcpdump to capture packets on the minio server.

Any other thoughts are appreciated.

davegold · September 10, 2017, 12:54am

I figured this out, want to document it for the sake of others.

Synopsis:
When using Duplicati with a Windows service, have to specify TLS12 at the job level in order to work with Minio. Setting at a global level does not appear to take.
It did automagically work when I was using it as a desktop application instead of a Windows service, but I probably won’t have time to reproduce with a new system to determine the difference.

Details:
I used ssldump on my minio server to capture packets.
1.2.3.4 is the Duplicati client, 5.6.7.8 is the Minio server
The command I used on the Minio server was:
ssldump -k private.key -i ens160 -AnHT host 1.2.3.4

The main item I noticed was that minio was throwing an error:

ERRO[1775] TLS handshake failed with new connection 1.2.3.4:50736 at server 5.6.7.8:50002  cause=tls: client offered an unsupported, maximum protocol version of 302 source=[listener.go:172:github.com/minio/minio/pkg/http.(*httpListener).start.func2()]

Translation of this is that Minio was receiving a request for an unsupported TLS protocol number 302.
TLS Version and SSLDUMP helped me translate 302 to TLS 1.1

I was able to see this in the ssldump output, where 3.2 is TLS 1.1:

New TCP connection #7: 1.2.3.4(50736) <-> 5.6.7.8(50002)
TCP: 1.2.3.4(50736) -> 5.6.7.8(50002) Seq 3280480600.(103) ACK 34573                                                                                        17151 PUSH
7 1  0.0067 (0.0067)  C>SV3.2(98)  Handshake
      ClientHello
        Version 3.2
...snip...
7 2  0.0074 (0.0007)  S>CV3.1(2)  Alert
    level           fatal
    value           protocol_version

I then modified the backup job and set the allowable SSL versions to TLS12 only.
I found that I had to go through and save the job, then go back in and test, or it would still use the default settings (TLS 1.0 or 1.1).

The ssldump output then shows TLS 1.2:

New TCP connection #17: 1.2.3.4(51038) <-> 5.6.7.8(50002)
TCP: 1.2.3.4(51038) -> 5.6.7.8(50002) Seq 723427621.(155) ACK 1395876959 PUSH
17 1  0.0096 (0.0096)  C>SV3.3(150)  Handshake
      ClientHello
        Version 3.3
...snip...
TCP: 1.2.3.4(51038) -> 5.6.7.8(50002) Seq 723427926.(661) ACK 1395878819 PUSH
17 11 0.0481 (0.0017)  C>SV3.3(656)  application_data

Once I saw the “application_data” output, that indicated that the TLS1.2 connection was up, and the application was now working.

The only other thing to note is that when I was testing using an IP number for the target instead of a hostname, the test failed but threw valid output (meaning it popped up with an error message). That output was easy for my workaround, since it noted that there was a hash error. I was able to add that to the job, save the job, retest and it worked.
What I saw in that case was Minio with the output:

ERRO[3066] Error in reading from new TLS connection 1.2.3.4:51025 at server 5.6.7.8:50002  cause=EOF source=[listener.go:188:github.com/minio/minio/pkg/http.(*httpListener).start.func2()]

and ssldump showing that the client ended (“FIN”) the conversation:

TCP: 1.2.3.4(50848) -> 5.6.7.8(50002) Seq 4210990031.(0) ACK 3967959489 FIN
9    0.0872 (0.0289)  C>S  TCP FIN
TCP: 5.6.7.8(50002) -> 1.2.3.4(50848) Seq 3967959489.(53) ACK 4210990032 PUSH
9 11 0.0877 (0.0005)  S>CV3.3(48)  Alert

Hopefully this will be useful for others replacing Crashplan peer to peer with Duplicati+Minio.

–Dave

JonMikelV · September 10, 2017, 3:05am

davegold, glad to hear you go Minio working!

I myself have yet to figure out why it’s not working for me which is why I’m wonder if perhaps @kenkendk would consider making a “How-To” or “Tutorials” Category in which you could document the settings (and issues) associated with getting Duplicati working with Minio.

dgcom · September 10, 2017, 3:39am

@davegold Glad you made it work.
Looks like Minio wants only TLS 1.2 and Duplicati did not start negotiating it by default…
It should be possible to let Minio to accept lower TLS versions for simplicity, but having only TLS 1.2 supported is good for security.
Interesting, from my experience, successful SSL deployment depends on too many factors

BTW, what OS the client is running on? On Windows, .Net apps usually use Windows defaults for negotiating SSL version…

belidzs · September 11, 2017, 10:03am

I’m just here to confirm that @davegold’s suggestion works. Thanks for digging down to the bottom of this issue!

kenkendk · September 11, 2017, 1:04pm

Yes I would really like that!

EDIT: say hello to How-To

davegold · September 11, 2017, 6:00pm

@dgcom Windows 7 32-bit is my client OS.

Credit goes to DIY cloud backup: Installing and configuring the server - Intermittent Technology for the basics on setting up Duplicati and Minio.

topeju · March 6, 2018, 8:55pm

This solution (restricting Duplicati to use TLS12) has worked just fine for me so far. However, I was stupid enough to update my minio, which broke connectivity from Duplicati running as a Windows service again.

I suspect this is due to this pull request to minio, which restricts the TLS cipher suites that minio server supports.

Has anyone else run across this, and have you got this working?

JonMikelV · March 7, 2018, 5:36am

Yikes, that sounds bad.

If somebody (like @kenkendk, @kees-z, or @Pectojin) can confirm Duplicati doesn’t currently support the cipher suites the update limits Minio to using then we may want to move this to its own topic…

topeju · March 7, 2018, 6:18am

I investigated this a bit more and the Windows 10 computers I have might work just fine (two use Duplicati as a service, and one doesn’t), so this is only limited to the two Windows Server 2012 R2 computers I have.

(I do have some problems with backing up to one of the two Minio instances I have set up, but one of them is working, and both were updated to the latest version (2018-02-09T22:40:05Z.)

kenkendk · March 7, 2018, 8:08am

On Windows, Duplicati is running with .Net, which in turn uses the Windows SSL configuration.

In other words, if IE/Edge works with the server, so should Duplicati.

topeju · March 7, 2018, 9:21am

I checked IE on one of the Windows Server 2012 R2 computers, and it doesn’t connect to the Minio instance either, it just reports

Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to [https URL to the minio server] again. If this error persists, it is possible that this site uses an unsupported protocol or cipher suite such as RC4 (link for the details), which is not considered secure

(TLS 1.0, TLS 1.1, and TLS 1.2 are enabled in IE settings)

Checking the Minio using Firefox, it reports TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 is used with TLS 1.2, and it looks like Windows (2012R2, that is) doesn’t support anything with ChaCha20. However, I’d expect TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 to work as both TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 and TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521 are listed under TLS 1.2 ECC GCM cipher suites in Windows group policy management, and the SSL Cipher Suite Order isn’t configured so it should use the default ones.

topeju · March 7, 2018, 4:19pm

It looks like the cipher suites supported by Minio are not compatible with Windows 8.1/Server 2012R2, if the private key is RSA. However, an ECDSA key is compatible, as then Minio can use the TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 cipher suite, which is also supported by these Windows versions.

Based on a discussion I’ve had with aead in the PR referenced above, the Minio documentation will soon be updated to describe how to generate ECDSA keys, but in brief, the command is openssl ecparam -genkey -name prime256v1 | openssl ec -out private.key.

JonMikelV · March 7, 2018, 8:08pm

Thanks for all that research!

It sounds like it’s a general Minio / .NET issue, not something specific that Duplicati needs to deal with.

topeju · March 8, 2018, 4:24am

Yes, I agree, this is not a Duplicati issue like it first seemed.

JonMikelV · March 8, 2018, 3:46pm

Thanks for confirming (agreeing with?) my suspicion - and starting this topic in case others run into the issue.

louis · April 2, 2018, 1:46am

I hate to throw a wrench in this, but I could use some help…and I think this might actually be a Duplicati issue.

I have a Minio instance with a self-signed certificate. With Duplicati (2.0.2.1_beta_2017-08-01) installed headless on Debian 9 (also tried Ubuntu 17.04), I can’t connect to Minio even with these three advanced options set:

accept-any-ssl-certificate
s3-ext-forcepathstyle
allowed-ssl-versions Tls12 only

The kicker is I CAN get it working from the same version of Duplicati running on Windows 10. I even tried exporting the exact working configuration from Windows 10 and importing it into the Debian install…My next step is to try an ECDSA key on Debian unless anyone has better ideas.

Thanks for your time!

louis · April 2, 2018, 6:56pm

Aha! I’ve solved this!

I got it working with the new beta (2.0.3.3) and an RSA key by updating Mono from the source instead of the Debian repositories (Download - Stable | Mono).

I’ve also discovered that the hash Duplicati is looking for when accepting SSL certificates by hash is the SHA1 hash with colons removed.

Hope this helps!