Windows service + minio with ssl error


#4

I figured this out, want to document it for the sake of others.

Synopsis:
When using Duplicati with a Windows service, have to specify TLS12 at the job level in order to work with Minio. Setting at a global level does not appear to take.
It did automagically work when I was using it as a desktop application instead of a Windows service, but I probably won’t have time to reproduce with a new system to determine the difference.

Details:
I used ssldump on my minio server to capture packets.
1.2.3.4 is the Duplicati client, 5.6.7.8 is the Minio server
The command I used on the Minio server was:
ssldump -k private.key -i ens160 -AnHT host 1.2.3.4

The main item I noticed was that minio was throwing an error:

ERRO[1775] TLS handshake failed with new connection 1.2.3.4:50736 at server 5.6.7.8:50002  cause=tls: client offered an unsupported, maximum protocol version of 302 source=[listener.go:172:github.com/minio/minio/pkg/http.(*httpListener).start.func2()]

Translation of this is that Minio was receiving a request for an unsupported TLS protocol number 302.
TLS Version and SSLDUMP helped me translate 302 to TLS 1.1

I was able to see this in the ssldump output, where 3.2 is TLS 1.1:

New TCP connection #7: 1.2.3.4(50736) <-> 5.6.7.8(50002)
TCP: 1.2.3.4(50736) -> 5.6.7.8(50002) Seq 3280480600.(103) ACK 34573                                                                                        17151 PUSH
7 1  0.0067 (0.0067)  C>SV3.2(98)  Handshake
      ClientHello
        Version 3.2
...snip...
7 2  0.0074 (0.0007)  S>CV3.1(2)  Alert
    level           fatal
    value           protocol_version

I then modified the backup job and set the allowable SSL versions to TLS12 only.
I found that I had to go through and save the job, then go back in and test, or it would still use the default settings (TLS 1.0 or 1.1).

The ssldump output then shows TLS 1.2:

New TCP connection #17: 1.2.3.4(51038) <-> 5.6.7.8(50002)
TCP: 1.2.3.4(51038) -> 5.6.7.8(50002) Seq 723427621.(155) ACK 1395876959 PUSH
17 1  0.0096 (0.0096)  C>SV3.3(150)  Handshake
      ClientHello
        Version 3.3
...snip...
TCP: 1.2.3.4(51038) -> 5.6.7.8(50002) Seq 723427926.(661) ACK 1395878819 PUSH
17 11 0.0481 (0.0017)  C>SV3.3(656)  application_data

Once I saw the “application_data” output, that indicated that the TLS1.2 connection was up, and the application was now working.

The only other thing to note is that when I was testing using an IP number for the target instead of a hostname, the test failed but threw valid output (meaning it popped up with an error message). That output was easy for my workaround, since it noted that there was a hash error. I was able to add that to the job, save the job, retest and it worked.
What I saw in that case was Minio with the output:

ERRO[3066] Error in reading from new TLS connection 1.2.3.4:51025 at server 5.6.7.8:50002  cause=EOF source=[listener.go:188:github.com/minio/minio/pkg/http.(*httpListener).start.func2()]

and ssldump showing that the client ended (“FIN”) the conversation:

TCP: 1.2.3.4(50848) -> 5.6.7.8(50002) Seq 4210990031.(0) ACK 3967959489 FIN
9    0.0872 (0.0289)  C>S  TCP FIN
TCP: 5.6.7.8(50002) -> 1.2.3.4(50848) Seq 3967959489.(53) ACK 4210990032 PUSH
9 11 0.0877 (0.0005)  S>CV3.3(48)  Alert

Hopefully this will be useful for others replacing Crashplan peer to peer with Duplicati+Minio.

–Dave


Setting up email via web-interface not working SOLVED
Duplicati failed to connect Amazone s3
#5

davegold, glad to hear you go Minio working!

I myself have yet to figure out why it’s not working for me which is why I’m wonder if perhaps @kenkendk would consider making a “How-To” or “Tutorials” Category in which you could document the settings (and issues) associated with getting Duplicati working with Minio. :slight_smile:


#6

@davegold Glad you made it work.
Looks like Minio wants only TLS 1.2 and Duplicati did not start negotiating it by default…
It should be possible to let Minio to accept lower TLS versions for simplicity, but having only TLS 1.2 supported is good for security.
Interesting, from my experience, successful SSL deployment depends on too many factors :slight_smile:

BTW, what OS the client is running on? On Windows, .Net apps usually use Windows defaults for negotiating SSL version…


#7

I’m just here to confirm that @davegold’s suggestion works. Thanks for digging down to the bottom of this issue!


#8

Yes I would really like that!

EDIT: say hello to #howto


Setting up self-hosted Minio on Windows 10 with automatically renewing SSL certificate
#9

@dgcom Windows 7 32-bit is my client OS.

Credit goes to DIY cloud backup: Installing and configuring the server - Intermittent Technology for the basics on setting up Duplicati and Minio.


#10

This solution (restricting Duplicati to use TLS12) has worked just fine for me so far. However, I was stupid enough to update my minio, which broke connectivity from Duplicati running as a Windows service again.

I suspect this is due to this pull request to minio, which restricts the TLS cipher suites that minio server supports.

Has anyone else run across this, and have you got this working?


#11

Yikes, that sounds bad.

If somebody (like @kenkendk, @kees-z, or @Pectojin) can confirm Duplicati doesn’t currently support the cipher suites the update limits Minio to using then we may want to move this to its own topic…


#12

I investigated this a bit more and the Windows 10 computers I have might work just fine (two use Duplicati as a service, and one doesn’t), so this is only limited to the two Windows Server 2012 R2 computers I have.

(I do have some problems with backing up to one of the two Minio instances I have set up, but one of them is working, and both were updated to the latest version (2018-02-09T22:40:05Z.)


#13

On Windows, Duplicati is running with .Net, which in turn uses the Windows SSL configuration.

In other words, if IE/Edge works with the server, so should Duplicati.


#14

I checked IE on one of the Windows Server 2012 R2 computers, and it doesn’t connect to the Minio instance either, it just reports

Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to [https URL to the minio server] again. If this error persists, it is possible that this site uses an unsupported protocol or cipher suite such as RC4 (link for the details), which is not considered secure

(TLS 1.0, TLS 1.1, and TLS 1.2 are enabled in IE settings)

Checking the Minio using Firefox, it reports TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 is used with TLS 1.2, and it looks like Windows (2012R2, that is) doesn’t support anything with ChaCha20. However, I’d expect TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 to work as both TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 and TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521 are listed under TLS 1.2 ECC GCM cipher suites in Windows group policy management, and the SSL Cipher Suite Order isn’t configured so it should use the default ones.


#15

It looks like the cipher suites supported by Minio are not compatible with Windows 8.1/Server 2012R2, if the private key is RSA. However, an ECDSA key is compatible, as then Minio can use the TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 cipher suite, which is also supported by these Windows versions.

Based on a discussion I’ve had with aead in the PR referenced above, the Minio documentation will soon be updated to describe how to generate ECDSA keys, but in brief, the command is openssl ecparam -genkey -name prime256v1 | openssl ec -out private.key.


#16

Thanks for all that research!

It sounds like it’s a general Minio / .NET issue, not something specific that Duplicati needs to deal with.


#17

Yes, I agree, this is not a Duplicati issue like it first seemed.


#18

Thanks for confirming (agreeing with?) my suspicion - and starting this topic in case others run into the issue.


#19

I hate to throw a wrench in this, but I could use some help…and I think this might actually be a Duplicati issue.

I have a Minio instance with a self-signed certificate. With Duplicati (2.0.2.1_beta_2017-08-01) installed headless on Debian 9 (also tried Ubuntu 17.04), I can’t connect to Minio even with these three advanced options set:

  • accept-any-ssl-certificate
  • s3-ext-forcepathstyle
  • allowed-ssl-versions Tls12 only

The kicker is I CAN get it working from the same version of Duplicati running on Windows 10. I even tried exporting the exact working configuration from Windows 10 and importing it into the Debian install…My next step is to try an ECDSA key on Debian unless anyone has better ideas.

Thanks for your time!


#20

Aha! I’ve solved this!

I got it working with the new beta (2.0.3.3) and an RSA key by updating Mono from the source instead of the Debian repositories (Download - Stable | Mono).

I’ve also discovered that the hash Duplicati is looking for when accepting SSL certificates by hash is the SHA1 hash with colons removed.

Hope this helps!


#21

Hi - I have Minio with SSL on a Windows 10 machine. I used the excellent guide I found on this forum (thanks).

I am running Duplicati 2.0.3.3. On a windows 10 machine (different one from where minio is running), the connection is fine. However, on a windows 7 machine - with the exact same configuration - I get the error “A WebException with status SecureChannelFailure was thrown.”

I have confirmed that tls 1.2 is enabled on the windows 7 machine, and have restricted the allowed ssl to only tls 1.2, as instructed. Nevertheless… nothing.

This makes me think it is something with Windows 7 & the latest version of duplicati? Anyone have any guesses? Anyone know where I should be looking on the windows 7 computer for duplicati logs? I’m afraid my windows 7 specific knowledge is pretty low (but I do have a couple of these I need to back up).

Thank you for any help you can provide - sorry for replying to an older thread, but it seemed on topic.


#22

Ah, sorry - using kenkend’s suggestion to try connecting to minio through IE instead of Chrome (and without involving duplicati), I indeed found the issue is with Windows SSL configuration.

So, the same issue causing problems for Windows 2012 in the comments above appears to be the problem for me for Windows 7. I will investigate the solution topeju suggests.


#23

I see a similar issue.

I installed Minio on a local linux server with a valid certificate. I originally installed Duplicati the regular non-service way. This worked as expected. I had to use the accept any certificate if I used a LAN IP instead of the FQDN, but otherwise no special options. I exported the backup config, moved to using Duplicati as a service, and imported the backup config. I started getting the same error as originally described in this thread.

The Minio HTTPS page works without issue in chrome and edge. Chrome confirmed TLS 1.2 was in use. In my case moving Duplicati to a service was the only change that made this issue appear. Manually forcing TLS1.2 in the Duplicati options seems to work. Does anyone have any additional insight on this?