Error after updating to Ubuntu Server 22.04 - Failed: EVP_MD_size assembly:<unknown assembly> type:<unknown type> member:(null)

dhollingsworth · May 8, 2022, 4:56pm

Hi,

After upgrading to Ubuntu Server 22.04 LTS I now receive the following error for all back ups:

Failed: EVP_MD_size assembly: type: member:(null)

Log file details follows:
Failed: EVP_MD_size assembly: type: member:(null)
Details: System.EntryPointNotFoundException: EVP_MD_size assembly: type: member:(null)
at (wrapper managed-to-native) FasterHashing.InteropOpenSSL11_libcrypto_dll.EVP_MD_size(intptr)
at FasterHashing.OpenSSL11_libcrypto_dll_HashAlgorithmSHA256…ctor () [0x0003b] in <787629f3dc434608a1b2bf412946d433>:0
at FasterHashing.OpenSSL11_libcrypto_dll_HashAlgorithm.Create (System.String name) [0x00036] in <787629f3dc434608a1b2bf412946d433>:0
at FasterHashing.OpenSSL11HashAlgorithm.Create (System.String algorithm, FasterHashing.OpenSSL11HashAlgorithm+LibraryName library) [0x0003a] in <787629f3dc434608a1b2bf412946d433>:0
at FasterHashing.FasterHash.Create (System.String algorithm, System.Boolean allowfallback, FasterHashing.HashImplementation implementation) [0x00034] in <787629f3dc434608a1b2bf412946d433>:0
at Duplicati.Library.Utility.HashAlgorithmHelper.Create (System.String name) [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0
at Duplicati.Library.Main.Options.get_BlockhashSize () [0x00025] in :0
at Duplicati.Library.Main.Volumes.VolumeBase…ctor (Duplicati.Library.Main.Options options) [0x0002b] in :0
at Duplicati.Library.Main.Volumes.VolumeWriterBase…ctor (Duplicati.Library.Main.Options options, System.DateTime timestamp) [0x00000] in :0
at Duplicati.Library.Main.Volumes.FilesetVolumeWriter…ctor (Duplicati.Library.Main.Options options, System.DateTime timestamp) [0x00000] in :0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01048] in :0
at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in :0
at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011c] in :0

Log data:
2022-05-08 16:14:04 +00 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.EntryPointNotFoundException: EVP_MD_size assembly: type: member:(null)
at (wrapper managed-to-native) FasterHashing.InteropOpenSSL11_libcrypto_dll.EVP_MD_size(intptr)
at FasterHashing.OpenSSL11_libcrypto_dll_HashAlgorithmSHA256…ctor () [0x0003b] in <787629f3dc434608a1b2bf412946d433>:0
at FasterHashing.OpenSSL11_libcrypto_dll_HashAlgorithm.Create (System.String name) [0x00036] in <787629f3dc434608a1b2bf412946d433>:0
at FasterHashing.OpenSSL11HashAlgorithm.Create (System.String algorithm, FasterHashing.OpenSSL11HashAlgorithm+LibraryName library) [0x0003a] in <787629f3dc434608a1b2bf412946d433>:0
at FasterHashing.FasterHash.Create (System.String algorithm, System.Boolean allowfallback, FasterHashing.HashImplementation implementation) [0x00034] in <787629f3dc434608a1b2bf412946d433>:0
at Duplicati.Library.Utility.HashAlgorithmHelper.Create (System.String name) [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0
at Duplicati.Library.Main.Options.get_BlockhashSize () [0x00025] in :0
at Duplicati.Library.Main.Volumes.VolumeBase…ctor (Duplicati.Library.Main.Options options) [0x0002b] in :0
at Duplicati.Library.Main.Volumes.VolumeWriterBase…ctor (Duplicati.Library.Main.Options options, System.DateTime timestamp) [0x00000] in :0
at Duplicati.Library.Main.Volumes.FilesetVolumeWriter…ctor (Duplicati.Library.Main.Options options, System.DateTime timestamp) [0x00000] in :0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x001bb] in :0

Andor_Kiss · May 11, 2022, 11:49am

Same here; I believe it’s an OpenSSL related bug.

gpatel-fr · May 11, 2022, 12:01pm

Ubuntu 22.04 does not support Openssl 1.1. All software must be upgraded to use Openssl 3.0.

foxhollow · May 11, 2022, 7:14pm

Same issue with Fedora 36. Works just fine with F35, but when I upgraded to F36 this error occured. I had to revert to F35 until this issue is resolved.

Andor_Kiss · May 11, 2022, 7:51pm

Workaround here: boot - Upgrading to Ubuntu 22.04 causes libcrypto errors, apt/dpkg broken - Ask Ubuntu

Tested and works in Pop_OS 22.04 LTS (and likely Ubuntu 22.04).

ts678 · May 11, 2022, 8:39pm

Welcome to the forum @foxhollow

Does Fedora (unlike Ubuntu) make it easy to add libssl1.1 to supplement latest release?
You wouldn’t want two copies of the openssl command, but I think libaries are versioned.

https://repology.org/project/openssl/versions reveals which distros moved to OpenSSL 3.
I know Fedora is kind of leading edge, but I’m surprised some rolling distros didn’t move.
I’m also surprised that it looks like Ubuntu first let loose on an LTS release. Seems risky.

Thanks for the workaround and the test. This mention should link here from GitHub issue.

OneDrive backup error message: EVP_MD_size assembly:<unknown assembly> type:<unknown type> member:(null) #4716

Anybody here a developer? Please look over https://github.com/kenkendk/FasterHashing.

2.0.2.11_canary_2017-10-20

Updated FasterHashing, so it automatically picks up openssl 1.0 and 1.1 on all platforms

Possibly it needs to be taught about OpenSSL 3, but I’m surprised it doesn’t fall back safely:

falling back to the managed implementation if none are found.

as documented, after Supported libraries (OpenSSL 1.0 and 1.1 API) are found unavailable.
An open question needing test and investigation is whether the version 3 API is compatible.

Actually getting FasterHashing changed (if that is required) requires its owner’s scarce help.
Maybe a less elegant temporary workaround to at least keep running (slower) can be found.

Andor_Kiss · May 11, 2022, 8:56pm

Well, I’m pretty sure (and someone please correct me if I’m wrong) that the libraries are versioned - so there shouldn’t be any issues with OpenSSL 3.0 moving forward. I guess it’s a temporary solution and the proper one is a re-compile and rebuild against OpenSSL 3.0.

ts678 · May 11, 2022, 11:06pm

I’m not a C# developer but AFAIK this is in managed code, meaning it wouldn’t link openssl at compile.
The proper solution is maybe to look into how FasterHashing searches for libraries, then update that…
That’s hindered by not having access to do anything with that project without help from its maintainer…

I do notice in the code (now that I look a bit) that one can disable probing. I wonder if it can be forced quietly into managed code using some of the below. Or maybe that’s where we got, and it’s broken…

github.com

kenkendk/FasterHashing/blob/6198adf6a8af08dc3c69502693b2730d8736f85f/FasterHashing/FasterHash.cs#L171-L199


      
          // Finally test for OpenSSL versions, newest first
          string version = null;
          if (string.Equals(Environment.GetEnvironmentVariable("FH_DISABLE_OPENSSL11"), "1", StringComparison.OrdinalIgnoreCase))
          {
              System.Diagnostics.Trace.WriteLine("OpenSSL 1.1 disabled, not probing");
          }
          else
          {
              version = OpenSSL11Version;
              if (version != null)
              {
                  System.Diagnostics.Trace.WriteLine($"Found OpenSSL 1.1 library with version string: {version}");
                  return HashImplementation.OpenSSL11;
              }
          }
          
          if (string.Equals(Environment.GetEnvironmentVariable("FH_DISABLE_OPENSSL10"), "1", StringComparison.OrdinalIgnoreCase))
          {
              System.Diagnostics.Trace.WriteLine("OpenSSL 1.0 disabled, not probing");
          }

This file has been truncated. show original

I think this is correct. I can run ldd to print traditional shared lib dependencies for my old openssl version:

> which openssl
/usr/bin/openssl
> ldd /usr/bin/openssl
	linux-vdso.so.1 (0x00007fff039ff000)
	libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007ffb19582000)
	libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007ffb190b7000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffb18e98000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffb18aa7000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffb188a3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffb19ac2000)
> dpkg -S /usr/lib/x86_64-linux-gnu/libssl.so.1.1
libssl1.1:amd64: /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> dpkg -S /usr/bin/openssl
openssl: /usr/bin/openssl
>

Above is Linux Mint 19.3, a Ubuntu derivative. So lib installs probably don’t clash (and you proved it).
The openssl command itself looks like it’s probably in the same path-friendly place for all its versions.

Still looking for a developer volunteer to sort things out. Developer and other volunteers in code, test, documentation, forum, and other areas are always highly valuable to maintain and improve Duplicati.

foxhollow · May 12, 2022, 5:53am

Hello ts678,

Does Fedora (unlike Ubuntu) make it easy to add libssl1.1 to supplement latest release?
You wouldn’t want two copies of the openssl command, but I think libaries are versioned.

This is beyond my Linux knowledge at this time.

This error is the first time I have had any issues with Duplicati, it has always been a solid piece of software which is my main backup solution. I was surprised when it didn’t work on Fedora which always just works in my experience.

gpatel-fr · May 12, 2022, 8:01am

easy as:
dnf install openssl1.1

Steven_Meyer · June 30, 2022, 12:42pm

Does anybody know if a fix for this is in the works? I’m not doing those work around due to the nature of the server.

linmanfu · July 1, 2022, 10:07am

The libssl problem is also being discussed over at GitHub Issues.

@dhollingsworth could you please change the post titles to something more appropriate, such as “Duplicati on Linux requires OpenSSL 1, which is not supported on recent Linux distros”? The long quoted error message “EVP_MD” makes it look like this issue involves some obscure set up, but we now know that this bug is a serious problem for the future of Duplicati on Linux.

dhollingsworth · July 2, 2022, 3:14pm

@linmanfu I’ve had a look, but for the life of me, I can’t see how to edit the title.

ts678 · July 2, 2022, 4:53pm

I hoped you (as original poster) would have an edit icon to its right, but if not, say what you want it to be, although personally I think the challenge will be more in attracting someone to pursue this, not topic title.

Eye-catching titles don’t help if no skilled eyes are proactively looking about for problems needing fixing. Developers might be more prone to look at GitHub anyway. It’s a true issue tracker, but still rather weak.

Duplicati community at the moment seems highly short on development volunteers, and it’s unfortunate.
As a community effort, all progress comes from volunteers in any area, including code, docs, forum. etc.

drwtsn32 · July 3, 2022, 3:12pm

Is this problem limited to the OneDrive back end? I spun up a 22.04 VM and didn’t have issues using my normal TLS back end (WebDAVS).

ts678 · July 4, 2022, 11:42am

Was this a fresh install or an upgrade? My 20.10 VM said it was unsupported, so offered 21.10 which offered 22.04 which is asking me Remove obsolete packages? which includes libssl1.1. For now

$ dpkg -l 'libssl*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version           Architecture Description
+++-===============-=================-============-===============================================
ii  libssl1.1:amd64 1.1.1l-1ubuntu1.5 amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl3:amd64   3.0.2-0ubuntu1.5  amd64        Secure Sockets Layer toolkit - shared libraries

I would have thought that would have trouble. I think mono actually has a Let’s Encrypt-challenged old version of BoringSSL for its TLS, but Duplicati looks around elsewhere for something to do hash work.

github.com

duplicati/duplicati/blob/d716051cdca941f13c0060b6bcc9b5ac3a3f8ee5/Duplicati/Library/Utility/HashAlgorithmHelper.cs#L23-L38


      
          	/// <summary>

          	/// This class helps picking the fastest hash algorithm implementation,

              /// which is what <seealso cref="System.Security.Cryptography.HashAlgorithm.Create()"/> should do, but does not.

          	/// </summary>
          	public static class HashAlgorithmHelper
              {

                  /// <summary>

                  /// Create the hash algorithm with the specified name.

                  /// </summary>

                  /// <returns>The hash algorithm.</returns>

                  /// <param name="name">The name of the algorithm to create.</param>

                  public static HashAlgorithm Create(string name)

                  {

                      return FasterHashing.FasterHash.Create(name);

                  }
              }

might be where it connects to the below output from

mono --trace=FasterHashing Duplicati.GUI.TrayIcon.exe

which in order to work right needs

export AUTOUPDATER_Duplicati_SKIP_UPDATE=true

[0x7f83754d3640: 1.23579 0] ENTER: FasterHashing.FasterHash:Create (string,bool,FasterHashing.HashImplementation)([STRING:0x7f8399f80cc0:SHA256], 1, 0)
[0x7f83754d3640: 1.23581 1] ENTER: FasterHashing.FasterHash:get_PreferedImplementation ()()
[0x7f83754d3640: 1.23582 1] LEAVE: FasterHashing.FasterHash:get_PreferedImplementation ()result=2
[0x7f83754d3640: 1.23583 1] ENTER: FasterHashing.OpenSSL11HashAlgorithm:Create (string,FasterHashing.OpenSSL11HashAlgorithm/LibraryName)([STRING:0x7f8399f80cc0:SHA256], 0)
[0x7f83754d3640: 1.23584 2] ENTER: FasterHashing.OpenSSL11HashAlgorithm:get_DefaultLibrary ()()
[0x7f83754d3640: 1.23584 2] LEAVE: FasterHashing.OpenSSL11HashAlgorithm:get_DefaultLibrary ()result=2
[0x7f83754d3640: 1.23585 2] ENTER: FasterHashing.OpenSSL11_libssl_so_1_1_HashAlgorithm:Create (string)([STRING:0x7f8399f80cc0:SHA256])
[0x7f83754d3640: 1.23586 3] ENTER: FasterHashing.OpenSSL11_libssl_so_1_1_HashAlgorithmSHA256:.ctor ()(this:0x7f8399862310[FasterHashing.OpenSSL11_libssl_so_1_1_HashAlgorithmSHA256 Duplicati.GUI.TrayIcon.exe])
[0x7f83754d3640: 1.23587 4] ENTER: (wrapper managed-to-native) FasterHashing.InteropOpenSSL11_libssl_so_1_1:EVP_get_digestbyname (string)([STRING:0x7f8399f80cc0:SHA256])
[0x7f83754d3640: 1.23589 4] LEAVE: (wrapper managed-to-native) FasterHashing.InteropOpenSSL11_libssl_so_1_1:EVP_get_digestbyname (string)result=0x7f836a91c200
[0x7f83754d3640: 1.23590 4] ENTER: (wrapper managed-to-native) FasterHashing.InteropOpenSSL11_libssl_so_1_1:EVP_MD_size (intptr)(0x7f836a91c200)
[0x7f83754d3640: 1.23591 4] LEAVE: (wrapper managed-to-native) FasterHashing.InteropOpenSSL11_libssl_so_1_1:EVP_MD_size (intptr)result=32

$ pwd
/lib/x86_64-linux-gnu
$ ls -1 libssl*
libssl3.so
libssl.so.1.1
libssl.so.3
$ dpkg -S libssl3.so
thunderbird: /usr/lib/thunderbird/libssl3.so
libnss3:amd64: /usr/lib/x86_64-linux-gnu/libssl3.so
$ dpkg -S libssl.so.1.1
libssl1.1:amd64: /usr/lib/x86_64-linux-gnu/libssl.so.1.1
$ dpkg -S libssl.so.3
libssl3:amd64: /usr/lib/x86_64-linux-gnu/libssl.so.3
$ nm --dynamic libssl.so.1.1 | grep 'EVP_MD.*size'
                 U EVP_MD_size@OPENSSL_1_1_0
$ nm --dynamic libssl.so.3 | grep 'EVP_MD.*size'
                 U EVP_MD_get_size@OPENSSL_3.0.0
$

EVP_MD_size

The EVP_MD_type(), EVP_MD_nid(), EVP_MD_name(), EVP_MD_pkey_type(), EVP_MD_size(), EVP_MD_block_size(), EVP_MD_flags(), EVP_MD_CTX_size(), EVP_MD_CTX_block_size(), EVP_MD_CTX_type(), and EVP_MD_CTX_md_data() functions were renamed to include get or get0 in their names in OpenSSL 3.0, respectively. The old names are kept as non-deprecated alias macros.

but alias source macros don’t help binaries that expect the old names. I wonder if that’s our problem?

If anybody out there has a good understanding of mono Interop with Native Libraries, please assist us.

umgfoin · December 7, 2022, 8:15pm

A possible solution to openssl3 issues can be found here on github: This should ensure openssl3-support for FasterHashing.
++umgfoin.