Distro-agnostic build of System.Security.Cryptography.Native for musl-libc based systems

[am11]

Background

In GNU-libc based systems, if libssl is available with a different version than the few hard-coded ones:

https://github.com/dotnet/corefx/blob/5710b6d09441a0a2d3cb9778ae927da14b5087cd/src/Native/Unix/System.Security.Cryptography.Native/opensslshim.c#L50-L76

dotnet.exe gives an error from line 84: No usable version of libssl was found\rAborted when running the application dotnet-run command. At this point, ldd shows no errors (as expected):

ldd ~/dotnet.3.0.100-preview-009812/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Security.Cryptography.Native.OpenSsl.so
        linux-vdso.so.1 (0x00007ffd38e11000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007faf8f139000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007faf8ed7b000)
        /lib/ld-linux-x86-64.so.2 (0x00007faf8f558000)

Going by the opensslshim code, the version can be overridden using an environment variable. For example, if a distro has libssl.so.46 available:

dotnet new ...
CLR_OPENSSL_VERSION_OVERRIDE=46 dotnet run
# run again
CLR_OPENSSL_VERSION_OVERRIDE=46 dotnet run
...

# or export the variable once
export CLR_OPENSSL_VERSION_OVERRIDE=46
dotnet run
dotnet run # again
...

This implies that the product is compiled in a distro-agnostic manner for GNU-libc based systems, which is great!

Problem

The latest 3.0 preview build for musl-libc is not built in a distro-agnostic manner (FEATURE_DISTRO_AGNOSTIC_SSL was off on the official build machine it seems?), therefore the binary is linked to a specific version of crypto and SSL libs: libcrypto.so.1.0.0 and libssl.so.1.0.0 available on Alpine Linux but not any other musl-libc based systems.

Void Linux comes with both flavors, glibc and musl-libc. This independent distro provides libressl via package management (XBPS: xbps-install openssl) with /usr/lib/libssl.so.46 and /usr/lib/libcrypto.so.44.

To make Hello World work on glibc based Void Linux, export variable once: CLR_OPENSSL_VERSION_OVERRIDE=46. Full repro:

# Docker on Windows 10
# ~Enter the Void~
docker run -it voidlinux/voidlinux sh

# inside Void Linux
xbps-install -S
xbps-install bash gettext openssl curl icu

curl -O https://download.visualstudio.microsoft.com/download/pr/9f071c35-36b4-48c9-bcc2-b381ecb6cada/5be4784f19c28cb58f8c79219347201a/dotnet-sdk-3.0.100-preview-009812-linux-x64.tar.gz

mkdir ~/dotnet.3.0.100-preview-009812/
tar -xzvf dotnet-sdk-3.0.100-preview-009812-linux-x64.tar.gz -C ~/dotnet.3.0.100-preview-009812/

# create a new console app to get a pretty error:
~/dotnet.3.0.100-preview-009812/dotnet new console -n test-hello
# No usable version of libssl was found
# Aborted

# set export CLR_OPENSSL_VERSION_OVERRIDE to 46 and try again:
export CLR_OPENSSL_VERSION_OVERRIDE=46

~/dotnet.3.0.100-preview-009812/dotnet new console -n test-hello
# ... few minutes later (due to first time usage setup) ....
# it works!

cd test-hello
~/dotnet.3.0.100-preview-009812/dotnet run
# it works!
~/dotnet.3.0.100-preview-009812/dotnet run
# it works again!

Lets try the same thing in voidlinux/voidlinux-musl:

# Docker on Windows 10
# ~Enter the Void-musl~
docker run -it voidlinux/voidlinux-musl sh

# inside Void Linux
xbps-install -S
xbps-install bash gettext openssl curl icu

# same version but different URL for musl-x64
curl -O https://download.visualstudio.microsoft.com/download/pr/0a0422a4-af0d-4742-a4ee-65bf87f146f9/6cad8369e06b3246fee3feab6d0256da/dotnet-sdk-3.0.100-preview-009812-linux-musl-x64.tar.gz

mkdir ~/dotnet.3.0.100-preview-009812/
tar -xzvf dotnet-sdk-3.0.100-preview-009812-linux-musl-x64.tar.gz -C ~/dotnet.3.0.100-preview-009812/

# try to create a new console app:
~/dotnet.3.0.100-preview-009812/dotnet new console -n test-hello

instead it throws an exception:

System.TypeInitializationException: The type initializer for 'Crypto' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library libSystem.Security.Cryptography.Native.OpenSsl: No such file or directory
at Interop.Crypto.GetMaxMdSize()
at Interop.Crypto..cctor()
--- End of inner exception stack trace ---
at Interop.Crypto.EvpSha256()
at Internal.Cryptography.HashProviderDispenser.CreateHashProvider(String hashAlgorithmId)
at System.Security.Cryptography.SHA256.Implementation..ctor()
at System.Security.Cryptography.SHA256.Create()
at Microsoft.DotNet.Cli.Telemetry.Sha256Hasher.Hash(String text)
at Microsoft.DotNet.Cli.Telemetry.TelemetryCommonProperties.GetTelemetryCommonProperties()
at Microsoft.DotNet.Cli.Telemetry.Telemetry.InitializeTelemetry()

At this point, ldd also complains:

Error loading shared library libcrypto.so.1.0.0: No such file or directory (needed by ./dotnet.3.0.100-preview-009812/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Security.Cryptography.Native.OpenSsl.so)
Error loading shared library libssl.so.1.0.0: No such file or directory (needed by ./dotnet.3.0.100-preview-009812/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Security.Cryptography.Native.OpenSsl.so)

The workaround is to create a symlink for exact versions: libcrypto.so.1.0.0 and libssl.so.1.0.0:

# create symlinks and try again:
ln -s /usr/lib/libcrypto.so.44 /usr/lib/libcrypto.so.1.0.0
ln -s /usr/lib/libssl.so.46 /usr/lib/libssl.so.1.0.0

~/dotnet.3.0.100-preview-009812/dotnet new console -n test-hello
# ... few minutes later (due to first time usage setup) ....
# it works!
cd test-hello
~/dotnet.3.0.100-preview-009812/dotnet run
# it works!

Request

Please set FEATURE_DISTRO_AGNOSTIC_SSL in official musl-x64 build plan, so in either flavors of Void Linux, there is only one way to configure the libssl, i.e. by setting CLR_OPENSSL_VERSION_OVERRIDE.

[am11]

I have published VoidWSL (glibc) and VoidMuslWSL images for easier/alternate ways of testing in Void Linux on Windows 10 box. Download, extract, install^☩ and run Void.exe/VoidMusl.exe with aforementioned steps.

^☩ current limitaiton: package has to be in system drive to get registered with WSL.

cc @janvorli, @bartonjs

[janvorli]

@am11 so there is no openssl 1.0 package available for the void Linux? Or is there one and you prefer not to install it for some reason?
@bartonjs what would be the consequences of running with LibreSSL instead of OpenSSL? Does it sound ok or do you see any pitfalls?

[bartonjs]

what would be the consequences of running with LibreSSL instead of OpenSSL? Does it sound ok or do you see any pitfalls?

It might work, it might not. We don't support it.

That said, if it's compiled directly (non-portable) against the LibreSSL headers, then it should, theoretically, work. Unless we happen to be counting on a behavioral difference in the libraries (I don't know of any, but there might be some).

[am11]

@janvorli, Void Linux manager provides Libressl package only, .NET Core and ASP.NET Core apps are working fine in my Void Linux boxes (both glibc and musllibc). Though I have not run all CoreFX tests on the system.

The said SSL package upon installation creates symlinks /usr/lib/libssl.so.46 and /usr/lib/libcrypto.so.44, which System.Security.Cryptography picks up without any problems in case of glibc-based Void Linux. This is also aligned with your new OS documentation: https://github.com/dotnet/source-build/blob/37d025d93/Documentation/boostrap-new-os.md#development-libraries

However, in case of musl-based Void Linux, it throws runtime error because it can't find fixed/hardcoded version of .so: libssl.so.1.0.0 and libcrypto.so.1.0.0. To fix this, we have to manually create symlinks: /usr/lib/libssl.so.46 <-> /usr/lib/libssl.so.1.0.0 and /usr/lib/libcrypto.so.44 <-> /usr/lib/libcrypto.so.1.0.0.

I tried to figure out the reason in related code and it seems that this is because for glibc-Linux, the official CoreFX package is compiled in a distro-agnostic way (cmake -DFEATURE_DISTRO_AGNOSTIC_SSL ...) and in case of musl-libc, it is not.

[janvorli]

I tried to figure out the reason in related code and it seems that this is because for glibc-Linux, the official CoreFX package is compiled in a distro-agnostic way

Right, it is the reason. The intent of the manual override of the openssl version in the portable build is actually not to enable pulling in other kinds of SSL libraries, but rather to enable fixing cases where the OpenSSL shared library is named differently on some exotic distros.

However, since there is no OpenSSL package for the Void Linux, changing the official build to portable to enable using LibreSSL there sounds reasonable. By doing that, we would not open any new holes, as we do that for glibc based distros anyways. However, as @bartonjs said, using other SSL library than OpenSSL won't be considered supported and you may hit unexpected issues when using some of the crypto stuff.

[metaleap]

@am11

The said SSL package upon installation creates symlinks /usr/lib/libssl.so.46 and /usr/lib/libcrypto.so.44, which System.Security.Cryptography picks up without any problems in case of glibc-based Void Linux. [...] However, in case of musl-based Void Linux

I never had musl-Void, only ever standard (glibc) Void. However, should note that dotnet aborting with No usable version of libssl was found was the case for both 2.x and 3.x (preview) for me both back in May when I first tried (didn't notice the env override in thread-wall back then), and still now in Sep when I thought I'd give this another whirl.

Thought I'd mention it since parts of the thread seem to imply it's a musl thing, AFAICT it's a void/libressl thing. To save others reading time: the workaround with an ephemeral CLR_OPENSSL_VERSION_OVERRIDE=46 does the trick. Bit sub-wholesome though, have to set up a special alias for dotnet with that override going in now as one wouldn't want to set this env globally.

[am11]

In the first post, shell script code shown for glibc-based Void has CLR_OPENSSL_VERSION_OVERRIDE=46 with working hello world example. Third comment probably should have restated it:

The said SSL package upon installation creates symlinks /usr/lib/libssl.so.46 and /usr/lib/libcrypto.so.44, which System.Security.Cryptography with CLR_OPENSSL_VERSION_OVERRIDE=46 picks up without any problems in case of glibc-based Void Linux.

On the other hand, musl-libc variant -- at the time -- was having issue with hello world (even with that environment variable set) because corefx build for musl-libc was non-portable (favoring Alpine Linux). In non-portable builds, opensslshim.c does not get compiled at all, which reads the environment variable and other forms (libssl.so.1.1, libssl.so.1.0.0 etc.). PR fixed this issue by making musl-libc builds portable; which ultimately switches on -DFEATURE_DISTRO_AGNOSTIC_SSL arg passed to cmake causing it to compile with opensslshim.c: https://github.com/dotnet/corefx/blob/b994610edca1e1f8b08a0be5ab36dd4214578ae5/src/Native/build-native.sh#L391