Send-mail and --accept-any-ssl-certificate doesn't work with 2.0.4.1_experimental

Support
1

Hi,

I cannot send mail with 2.0.4.1_experimental. It works fine with old 2.0.3.6_canary. I tried to get more log information about why the http-options is not loaded but found nothing useful. I am sorry for czech locale, I tried to force english but it doesn’t work. Tested on Windows 7 64-bit. I have no idea what’s wrong.

Kind regards, Jan

Duplicati.CommandLine send-mail --send-mail-subject="Test" --send-mail-body="Body" --send-mail-to=my.email@email.ignore.cz --send-mail-url=smtp://smtp.domain.com:25 --accept-any-ssl-certificate --force-locale=en-US

Volba --accept-any-ssl-certificate není podporována protože modul Nastavit http
požadavky (http-options) nyní není načtený
Celá SMTP komunikace: Connected to smtp://mail-internal.prague.wood.cz:25/?start
tls=when-available

(SMTP communication ommitted)

Zprávu se nepodařilo odeslat: System.Security.Authentication.AuthenticationExcep
tion: The remote certificate is invalid according to the validation procedure.
 => The remote certificate is invalid according to the validation procedure.
2

Hello, can you try if --accept-specified-ssl-hash= also doesn’t work?

As workaround - if you have mailserver in LAN without valid cert, you can use:
--send-mail-url=smtp://mailserver:25/?starttls=never

1 Like
3

The starttls=never works. The accept-specified-ssl-hash has the same problem as above, module http-options is not loaded.

Thank you, Jan

4

hmm, I did not do it yet, but maybe this could work:
--enable-module http-options
https://duplicati.readthedocs.io/en/latest/06-advanced-options/#enable-module

5

No, I already tried --enable-module=http-options but it still says the module is not loaded. BTW this is core module, it should be loaded all the time. In my opinion the warning is misleading. At home I cannot use the starttls=never as I am also passing login/password to the mail server.

Due to this problem I cannot upgrade to latest experimental.

Thank you for your time and support, Jan

6

Well 2.0.4.2 is only an update to resolve a Docker container issue so unless you’re running Docker on your Windows 7 box you’re not missing anything in this update.

There were quite a lot of changes between 2.0.3.6 and 2.0.4.1 - have you confirmed the backup is running? It’s possible some other change is killing the backup before it gets to the email step.

If possible, consider creating a test job that just does a small local backup and sends an email…

7

All reports above are from Duplicati.CommandLine send-mail.

8

Sorry I missed that… well that rules out the GUI as the culprit I guess.

9

Thank you anyway. In my opinion the problem is that the core module http-options is not loaded - not sure how it can happen.

10

While the http-options module situation is interesting (and might be a fault – I don’t know that code enough), I think it only affects HTTPS, which isn’t in use here (or do you believe otherwise?). As for the SMTP transcript that was omitted, can I assume that action stopped after the STARTTLS, i.e. you didn’t get later error codes?

Email sends sometimes has advice about error codes (if yours got that far) and SSL/TLS debug (if it did not).

No mail report from scheduled jobs even though it works from send-mail/backup through web command line! has information about the email library that Duplicati uses, showing smtp equivalent to https -accept-any-ssl-certificate and explaining how this is not the same code either. Loosening certificate checking for SMTP does not appear possible short of avoiding SSL/TLS, however because you’re security-conscious and want TLS in use, note that proper certificate checking is also essential to avoiding things like man-in-the-middle intercept.

Maybe you can send Internet Explorer to https://mail-internal.prague.wood.cz:25 then check error messages.

How did you run this test without already upgrading, or are there different systems (if so, how do they differ)?

EDIT: Although I still think the secure way to do this is to debug the invalid certificate (rather than suppress it), testing on 2.0.3.3 shows that --accept-any-ssl-certificate did seem to convert the send-mail results away from an AuthenticationException complaining about the certificate (and you have to add --log-level=Error for some reason to see the message) into a timeout (not totally surprising because I wasn’t going to a real email server but to smtps://expired.badssl.com:443 because I know it will feed me an expired certificate that I can test with). Part of the current problem is that --accept-any-ssl-certificate is in the Advanced options dropdown section of “Configure http requests” although arguably such a concept is broadly applicable to any SSL/TLS connection and this raises a lot of implementation questions about different ways to set things, and scope of the settings. This level of design question doesn’t arise much in this user forum. Unfortunately GitHub issues didn’t raise it.

11

Thank you ts678 for your in-depth answer, I will try to address parts that I do understand.

The SMTP transcript ends with STARTTLS and 220 2.0.0. SMTP server ready. You are right that instead of ignoring the certificate I should verify at least the hash.

I finally decided to install the same Duplicati version at home and to my surprise, it can send emails with -accept-any-ssl-certificate. The same Windows 7 version, the same ESET antivirus. I mention antivirus because I suspect this is the reason why http-options module is not loaded. Most probably ESET blocked loading some underlying implementation DLL (which?). On the other hand suspending ESET does not help and on my home computer it works fine. Also the only message in ESET log I have found is that Duplicati.CommandLine.exe, Duplicati.Server.exe and Duplicati.Windows.Service.exe were sent to cloud for further investigation.

As there are no useful log messages of why built-in module http-options was not loaded, I tried to use ProcessMonitor but have found nothing useful. What could possibly fail in code of HttpOptions.cs?? Does the module loader catch exceptions thrown during module initialization and throws the error out of window instead of logging it?

Kind regards, Jan

12

How do you detect that it is not loaded? It is a core module and loaded by default.

That was my thought as well. The http-options module affects the System.Web classes, including the HttpWebRequest and FtpWebRequest. The email module is third-party, but I guess it does not use those. It probably creates the connection (unencrypted) and then enables TLS, which means that http-options does not have a chance to validate the certificate.

Can it also send without --accept-any-ssl-certificate set ?

13

Thank you Ken for you comments.

How do you detect that it is not loaded? It is a core module and loaded by default.

Yes, but see my initial post, is says in Czech language (není načtený = not loaded). I tried to use different locale but --force-locale=en-US did not worked. And --enable-module does not help.

Volba --accept-any-ssl-certificate není podporována protože modul 
Nastavit http
požadavky (http-options) nyní není načtený

Can it also send without --accept-any-ssl-certificate set ?

Nope, again see the initial post (it works only when I suppress TLS)

Zprávu se nepodařilo odeslat: 
System.Security.Authentication.AuthenticationExcep
tion: The remote certificate is invalid according to the validation procedure.

Thank you, Jan

14

For this, I meant on the machine where it works (“at home”)?

Ok, I google-translated it, and I see now it says the module is not loaded.

The module is actually dynamically loaded (even though it is a core module). You can try to add the option:

--console-log-level=explictonly

This should give you a number of error messages if the loading of the module fails.

15

On home computer it works with --accept-any-ssl-certificate without any problem and I need it as the server’s certificate is selfsigned (and I should use hash instead).

Thanks for --console-log-level=ExplicitOnly, we finally see something useful:

Chyba načtení sestavení C:\Program Files\Utils\Duplicati 2\Microsoft.WindowsAzur
e.Storage.dll, chybové hlášení: Could not load file or assembly ‘Microsoft.Data.
Services.Client, Version=5.8.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e
35’ or one of its dependencies. The located assembly’s manifest definition does
not match the assembly reference. (Exception from HRESULT: 0x80131040) => Could
not load file or assembly ‘Microsoft.Data.Services.Client, Version=5.8.1.0, Cult
ure=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The lo
cated assembly’s manifest definition does not match the assembly reference. (Exc
eption from HRESULT: 0x80131040)

Many thanks for your support, Jan

EDIT:
Could this be the reason (just the version is different): Could not load file or assembly 'Microsoft.Data.Services.Client, Version=5.5.0.0

16

That is just a problem getting the Azure storage backend to work, it should not affect the http-options module.

Could you try with --console-log-level=Profiling ? That should give a different set of error messages.

17

Unfortunately there is no log info prior the message that says option is not supported due to http-options module not loaded:

C:\Program Files\Utils\Duplicati 2>Duplicati.CommandLine send-mail --send-mail-s
ubject=“subject” --send-mail-body=“Body” --send-mail-to=myself@myself.cz --send
-mail-url=smtp://smtp.myslef.cz:25 --accept-any-ssl-certificate –
console-log-level=Profiling
Volba --accept-any-ssl-certificate není podporována protože modul Nastavit http
požadavky (http-options) nyní není načtený
Operace SendMail zahájena
Starting - Running SendMail
Running SendMail took 0:00:00:00.005
Celá SMTP komunikace: Connected to smtp://smtp.myself.cz:25/?start
tls=when-available
…

The only thing I can think of is that the http-options module crashed during initialization and thus is not available. Maybe error logging is missing for build in module initialization errors?

Jan

18

That is what the ExplictiOnly error messages should show. As you can see from the error message with Azure, it outputs any loading errors, so the http module errors should show there.

If there is a problem loading the module, you should not see it if you invoke “help”, as that will list all (readable) modules. Mine says:

> Duplicati.CommandLine.exe help

See duplicati.commandline.exe help <topic> for more information.
  General: example, changelog
  Commands: backup, find, restore, delete, compact, test, compare, purge,
  vacuum
  Repair: repair, affected, list-broken-files, purge-broken-files
  Debug: debug, logging, create-report, test-filters, system-info, send-mail
  Targets: amzcd, azure, b2, box, cloudfiles, dropbox, ftp, file, googledrive,
  gcs, hubic, jottacloud, mega, msgroup, onedrivev2, sharepoint, openstack,
  rclone, s3, ssh, od4b, mssp, sia, tahoe, webdav
  Modules: aes, gpg, zip, 7z, console-password-input, mssql-options,
  hyperv-options, http-options, sendhttp, sendmail, runscript, sendxmpp,
  check-mono-ssl
  Formats: date, time, size, encryption, compression
  Advanced: mail, advanced, returncodes, filter, filter-groups, <option>

You can see in Modules: it lists http-options. If it fails to load the dll, this module will not be present.

You can also try to read the help message from the module:

> Duplicati.CommandLine.exe help http-options
Configure http requests (http-options):
 This module exposes a number of properties that can be used to change the way
 http requests are issued
 Module is loaded automatically, use --disable-module to prevent this
 Supported options:
  --disable-expect100-continue (Boolean): Disable the expect header
...
19

Yes, module is listed and also shows configuration options.

Checked again at home and I was not correct - sending email at home works because my SMTP provider acquired valid certificate. When I want to use --accept-specified-ssl-hash it shows the same warning about module not loaded. In other words, I see same behavior at work and at home, just at home I am lucky to have valid certificate, internal SMTP server at work uses untrusted certificate.

I was trying to compare Process Monitor output between the two instances but now it doesn’t make sense.

Another idea - could the option be used before the build-in module is initialized?

Jan

20

In my opinion we can close this. Most probably one cannot send emails to SMTP server over TLS with no-valid server certificate. The http-options module options are not used for email - this was already pointed out in this thread.

Many thanks for your time, Jan