SSL certificates: figure out how libs/apps find them by default

[vcunat]

For some options already discussed see #8121 (comment).

[layus]

I found an excellent email exchange on the openssl mailing list: https://mta.openssl.org/pipermail/openssl-users/2015-December/002553.html.
Also, a most enlightening blog post at https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/.
And finally, a guix discussion about gnuTLS: https://lists.gnu.org/archive/html/guix-devel/2014-02/msg00245.html

Basically, we need

1. To define at NixOS level a "default trusted location" (currently /etc/ssl/certs/ca-certificates.crt) that can be hardcoded in every package requiring a trust store;
2. Ensure that all applications default to that location (Currently not the case for openssl, and possibly others. Openssl imparct Python for example.);
3. Allow users to specify a different trust store. (Currently using SSL_CERT_FILE, possibly other application-specific mechanisms.)

At the moment, openssl is broken without SSL_CERT_FILE because it has no default trust store. (Technically it is false, openssl trust store defaults to /nix/store/-openssl-version/etc/ssl/cert.pem, which does not exist 😈)
We therefore need to

• (option 1) Patch openssl to use /etc/ssl/certs/ca-certificates.crt as its default trust store;
• (option 2) Define SSL_CERT_FILE everywhere (including in systemd.globalEnvironment), to ensure that openssl finds its default store.
• (option 3) Make the ca-cert package a depedency of all the packages using it ?

Option 2 is straightforward, but we can do better than that!
Option 3 does not seem wise as we would trigger a mass rebuild for nothing.
I would advise to go for option 1 as a sane default.
With option 1, we need to fix openssl trust store, and get rid of the SSL_CERT_FILE infection (already 30 occurrences, including the valid ones.)
Regarding gnuTLS, we should add the option --with-default-trust-store-file (see #8121) as explained in the above mailing lists. GnuTLS provides no way to globally override this setting, but this is their design decision. Applications can always follow SSL_CERT_FILE if they want to.

So we need to ship openssl with sane defaults.
The main issue is that we want --openssldir to be /etc/ssl, but this is not possible because openssl also tries to install files there at build/install time. However, we can use the following trick (instead of reverse-engineering and patching the build process).

--- a/pkgs/development/libraries/openssl/default.nix
+++ b/pkgs/development/libraries/openssl/default.nix
@@ ... @@ stdenv.mkDerivation rec {
   configureFlags = [
     "shared"
     "--libdir=lib"
-    "--openssldir=etc/ssl"
+    "--openssldir=etc/ssl/openssl"
   ] ++ stdenv.lib.optionals withCryptodev [
     "-DHAVE_CRYPTODEV"
     "-DUSE_CRYPTODEV_DIGESTS"
   ];
@@ ... @@ stdenv.mkDerivation rec {
   postInstall = ''
     # If we're building dynamic libraries, then don't install static
     # libraries.
     if [ -n "$(echo $out/lib/*.so $out/lib/*.dylib $out/lib/*.dll)" ]; then
         rm "$out/lib/"*.a
     fi

     # remove dependency on Perl at runtime
     rm -r $out/etc/ssl/misc $out/bin/c_rehash
+
+    # Setup default ssl files location
+    # The openssl subdir is a trick to avoid infinite loops when building the environment without cacert.
+    mv $out/etc/ssl/openssl/* $out/etc/ssl
+    rmdir $out/etc/ssl/openssl
+    ln -s /etc/ssl/ $out/etc/ssl/openssl
+    ln -s /etc/ssl/certs/ca-certificates.crt $out/etc/ssl/cert.pem
   '';

@vcunat, Anything against this patch ?
If you agree, I will submit a pull-request.

[layus]

This is a small memo to understand how apps work with ssl files.
Obviously, the memo got bigger than expected :-)

Nix and NixOS are missing strong guidelines on how to find and manage the default trust store.
That store is the place where all the trusted root certificates are stored.

Due to this confusion, several issues were raised on this mailing list and on GitHub.

These issues concern

• NixOS: Programs working well in the shell, but failing within systemd units (services).
• Nix, without NixOS: Inconsistent certificate validity between applications in the nix store and "native" applications.

These issues arises because there is no consensus on where to find the default, system-wide trusted certificates.
This confusion affects all the distros likewise, and is not specific to Nix.

There are four ways that applications can use to locate the default trust database:

1. Trust the default location provided by their security library.
   OpenSSL has --openssldir and $SSL_CERT_FILE, GnuTLS has --with-default-trust-store-file, etc.

   For example, ArchLinux compiles OpenSSL with --openssldir=/etc/ssl, and provides "/etc/ssl/cert.pem" in the package ca-certificates-utils required by its base environment.
   Most linux distros will pick a trust store location and configure their packages to default to it.
   This default location is however not the same for everyone.

2. By looking at a set of common locations, like /etc/ssl/certs/ca-certificates.crt, /etc/ssl/certs/ca-bundle.crt, /etc/pki/tls/certs/ca-bundle.crt, etc.
   These paths are also provisioned by NixOS in the hope to catch programs looking for them.

3. Ask for and/or receive an explicit path, either using the environment or command line arguments.
   (OpenSSL honors -CAfile, curl has --cacert, etc.)

4. Ship its own trust store. (This is the case with Python's requests module and with Firefox for example.)

Most distros try to configure and patch the programs that use certificates to use their main trust store.
Nix however cannot provide packages with an hardcoded default store because nix packages are used on many different distros and even different systems.
On these systems, nix-profile.sh (installed in "~/.nix-profile/etc/profile.d/nix.sh")
tries to find the location of the system-wide trust store (using method 2. above), and configures SSL_CERT_FILE accordingly.

# Excerpt from nix-profile.sh.in :
# Set $SSL_CERT_FILE so that Nixpkgs applications like curl work.
if [ -e /etc/ssl/certs/ca-certificates.crt ]; then # NixOS, Ubuntu, Debian, Gentoo, Arch
    export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
elif [ -e /etc/ssl/certs/ca-bundle.crt ]; then # Old NixOS 
    export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt 
elif [ -e /etc/pki/tls/certs/ca-bundle.crt ]; then # Fedora, CentOS 
    export SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt 
elif [ -e "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" ]; then # fall back to cacert in Nix profile
    export SSL_CERT_FILE="$NIX_LINK/etc/ssl/certs/ca-bundle.crt"
elif [ -e "$NIX_LINK/etc/ca-bundle.crt" ]; then # old cacert in Nix profile 
    export SSL_CERT_FILE="$NIX_LINK/etc/ca-bundle.crt"
fi

This is extremely important for Nix itself to work properly as derivations and nix archives are downloaded using HTTPS.
By setting this environment variable which overrides the default location used by OpenSSL, Nix ensures that programs like cURL will be able to download archives securely.
We need standardization to avoid inconsistencies between applications
(why does curl fail on https://XXX.YY's certificate while firefox works flawlessy?)
and to provide a single location to update in case of issues with the root CA's.

As this variable only affects OpenSSL, the other security libraries and applications are left guessing for the right path to use and often fail.
They are unaware of SSL_CERT_FILE, and nix cannot compile them with a default trust store because there is no global sane value.
Therefore, nixpkgs sometimes merges patches that add support for SSL_CERT_FILE in crypto libraries.

On NixOS, the situation is completely reversed.
We have full control over the environment of the packages, and we know where the certificates are located.
We could even compile the packages to depend on cacert (the derivation providing the default trusted certificates).

However, making derivations depend on cacert is not a good idea because

1. NixOS is terrible at managing security updates.
   If one certificate needs to be revoked, then all the packages would require a rebuild.
2. Users would need to override the cacert package to provide custom trusted certificates,
   and hit the same mass rebuild issue.

For these reasons NixOS uses the path "/etc/ssl/certs/ca-certificates.crt" as the system's default trust store.
This path (as well as other common paths) is provisioned by the security/ca.nix module using files from the cacert derivation.
To ensure that packages find and use this path, NixOS exports SSL_CERT_FILE in the global environment.

However, this global environment is not available to systemd units, which is the reason why so many users hit the mailing list with the issue of a command working perfectly well in their shell and failing when started in a unit.
This situation is clearly a failure of NixOS and the Nix model where a derivation should "always work the same".
The easy fix on NixOS would be to add SSL_CERT_FILE to systemd.globalEnvironment, and have that variable added to all the systemd units.
The biggest downside is that changing (and adding) this variable will require a restart of all the units, even the ones not requiring that environment variable.

Now, on NixOS we do not need the flexibility to change the trust store, because we "know" that it will be "/etc/ssl/certs/ca-certificates.crt".

My proposal is therefore to make all the crypto libraries look for a default trust database in $SSL_CERT_FILE when defined, and in "/etc/ssl/certs/ca-certificates.crt" when not .
This way, NixOS would work flawlessly without defining SSL_CERT_FILE, because all the derivations would default on the right path.
And all the other Nix users would have the best experience by just setting SSL_CERT_FILE to the right location, or relying on "~/.nix-profile/etc/profile.d/nix.sh" to do it for them.

This proposal requires subtle patches to nixpkgs.
We need all the packages that provide a default store to default to "~/.nix-profile/etc/profile.d/nix.sh".
We should not however patch the packages that get their default store from another lib.

As an example, git needs no patching, because it relies on the default trust store provided by libcurl.
cURL however requires fixing, because it does not rely on the default trust store provided by openssl or its other backends.

The current lwp-protocol-https-cert-file.patch applied on the corresponding perl module fixes a non-issue:

Use $SSL_CERT_FILE to get the CA certificates.

diff -ru -x '*~' LWP-Protocol-https-6.02-orig/lib/LWP/Protocol/https.pm LWP-Protocol-https-6.02/lib/LWP/Protocol/https.pm
--- LWP-Protocol-https-6.02-orig/lib/LWP/Protocol/https.pm  2011-03-27 13:54:01.000000000 +0200
+++ LWP-Protocol-https-6.02/lib/LWP/Protocol/https.pm   2011-10-07 13:23:41.398628375 +0200
@@ -21,6 +21,11 @@
     }
     if ($ssl_opts{SSL_verify_mode}) {
    unless (exists $ssl_opts{SSL_ca_file} || exists $ssl_opts{SSL_ca_path}) {
+            if (defined $ENV{'SSL_CERT_FILE'}) {
+                $ssl_opts{SSL_ca_file} = $ENV{'SSL_CERT_FILE'};
+            }
+        }
+   unless (exists $ssl_opts{SSL_ca_file} || exists $ssl_opts{SSL_ca_path}) {
        eval {
        require Mozilla::CA;
        };

In this case, the Perl module does not rely on any default trust store, and requires the Mozilla::CA module when none is provided explicitly.
This patch changes the behavior of the Perl module. Any user of that module should provide a trust store explicitly or also depend on Mozilla::CA.

Finally, we may want to patch derivations providing their own trust store to use the system one, or not.
How can we ensure that the packages honor the default trust store if Firefox, Python's request module or Perl's Mozilla::CA bundle their own trusted certificates ?
Is this a security issue ?

[layus]

@edolstra, the previous message explains why we should not "kill" SSL_CERT_FILE, and why we should therefore revert the pull request #12748.

[edolstra]

Contrary to what the name of the PR implies, it does not kill SSL_CERT_FILE, just provide a default value. The only problem is that are a few programs (e.g. git, but apparently not curl) that don't respect SSL_CERT_FILE out of the box, so we may want to revert those parts of the PR.

[layus]

My issue with the current merged patch is that we removed the support of SSL_CERT_FILE from cURL and provided a default location for the trust store.
This means that the cURL on staging will not respect SSL_CERT_FILE and will only look into the provided default store ("/etc/ssl/certs/ca-certificates.crt"), as opposed to the old behaviour which respected SSL_CERT_FILE and looked in a set of known locations if SSL_CERT_FILE was not defined.
I was too eager to remove SSL_CERT_FILE on NixOS, and did not think about Nix users on other systems/OS.

I made more moderate patches on my own repo at https://github.com/layus/nixpkgs/tree/fix-openssl-defaults.
These patches include some of your own modifications, and respect the proposition made above to define the default location to NixOS's default and to respect SSL_CERT_FILE.

[zimbatm]

What if the default trust store was /nix/etc/ssl/cert.pem (or something under /nix) ? That's a path that we control on all systems. During install it could then be symlinked to the target OS's trust store. Before install it could also be pre-seeded with enough certs for nix to fetch stuff from the hydra cache.

On OSX the situation would be a bit more complicated because they store their certs in another format called the Keychain and it's all over the place. Homebrew's install of openssl solves the issue by generating their own cert.pem. I suppose that file needs to be kept in sync with the Keychain but again it could be put under /nix/etc/ssl/cert.pem.

The issue with relying on the SSL_CERT_FILE is that it might not be available when running sudo for example. On OSX, Homebrew is not able to fetch bottles (the pre-built binaries) when SSL_CERT_FILE is set. (but it works fine for source code for some reason). Homebrew uses /bin/curl to fetch. My guess is that's because Apple replaced curl's SSL with their own engine and forgot to remove the support of SSL_CERT_FILE.

[layus]

Looks like a great idea to me. You simply set the right symlink at install time. And it works equally well for Nix and NixOS. Perfect !

Does Windows support this ?

[zimbatm]

I haven't looked at Windows in details because it's not an official Nix target. Windows itself has it's own CA store like OSX and we would need an export script to generate the PEM file. It's possible that Cygwin and friends already deal with the issue. The day where we want to support Window I expect the target to be Windows+Cygwin or Windows+MinGW, ...

[vcunat]

What if the default trust store was /nix/etc/ssl/cert.pem (or something under /nix) ?

I'm not certain if someone actually wants to control the trust store globally for all nix stuff but separately from the OS. It seems enough to me to use the OS trust store as the default and allow overriding it by an env var.

BTW, rather than /nix/etc/ssl/ I would imagine /nix/var/nix/profiles/default/etc/ssl/.

[layus]

Using /nix/var/nix/profiles/default/etc/ssl/ might work if nix-env detects the location of the trust store.
However, this is not as flexible as /nix/etc/ssl/ which could be changed independently of the current environment.
Admitedly, there is no reason to change this once the /nix tree is installed.

[cleverca22]

also of note, setting SSL_CERT_FILE globally or per-service doesn't work in the case of php-fpm, it scrubs the env on startup by default, so it has to be entered into the php.ini file, or compiled in as a default

[nrolland]

@layus thank you it's really helpful to have a birdeye view to make sense of it..

[wmertens]

The real solution is to use the platform trust stores, but not all upstream projects support all platforms.

The second best solution is to periodically run something like https://github.com/raggi/openssl-osx-ca/blob/master/bin/openssl-osx-ca which dumps the trust store into a properly formatted file.

I am a big fan of storing global state in /nix/etc because, indeed, that is the one location under Nix control on all platforms. For backwards compatibility with existing installations, that can be a symlink to /etc.

So I propose that from now on all Nix-related global state is stored under /nix/etc, and that the Nix install script makes either a symlink to the trust store or runs a dump script for the platform and recommends running it periodically.

[magnetophon]

Any new on this?
I just got bitten by it too.

There's a couple of related issues, with various workarounds.
What is the best solution for now?

[layus]

Could you describe your specific issue, possibly in a new thread ? Link it to this thread, and tag me if needed. The current state should be OK for most use cases.

[4e6]

I want to bring up an issue when the user has custom root certificates installed. My setup is NixOS with extra certificates defined by security.pki.certificates config option.

Issue

There are two certificate bundles that are currently used in Nixpkgs:

• ${cacert}/etc/ssl/certs/ca-bundle.crt
• /etc/ssl/certs/ca-certificates.crt

Find usages:

[~/projects/nixos/nixpkgs]$ nix-shell -p ag --run 'ag /etc/ssl/certs/ca-bundle.crt pkgs'
[~/projects/nixos/nixpkgs]$ nix-shell -p ag --run 'ag /etc/ssl/certs/ca-certificates.crt pkgs'

The problem with the first, ${cacert}/etc/ssl/certs/ca-bundle.crt is that it doesn't contain extra certificates defined by NixOS config option security.pki.certificates. Currently they are assembled under the /etc/ssl/certs/ca-certificates.crt bundle.

The second approach, constant /etc/ssl/certs/ca-certificates.crt works perfectly with NixOS, but could have issues with Nix systems that use a different file for bundled certificates.

I experience problems with packages that use cacert bundle, ${cacert}/etc/ssl/certs/ca-bundle.crt, jdk and cargo in particular. I need to add postInstall step to openjdk8 package to reinstall correct certificates from /etc/ssl/certs/ca-certificates.crt. Wrapped cargo executable causes #8872 issue because of wrong certs file, but works fine unwrapped.

Proposed solution with managed bundle under the /nix directory should fix the issue.

[layus]

@4e6 You are right that ${cacert}/etc/ssl/certs/ca-bundle.crt should not be used. It forces a rebuild of the package everytime a certificate changes. That being said, /etc/ssl/certs/ca-certificates.crt should not be used directly, only as a fallback if NIX_SSL_CERT_FILE is not set. In all cases, NIX_SSL_CERT_FILE should be observed by nix packages.

This is the new standard as per NixOS/nix@fb2dd32. A migration has started with the new commit 942dbf8
Also, see NixOS/nix#921.

@edolstra We should probably make an announcement on the mailing list about how to manage certificates. Using NIX_SSL_CERT_FILE is quite a big change and may break many configurations.

[stale]

I marked this as stale due to inactivity. → More info

[mohe2015]

Still important to me

[rnhmjoj]

Finally #96763 made it into unstable. We now have all nss-based application using p11-kit and reading certificates from the system-wide trust store!

[sigprof]

Finally #96763 made it into unstable. We now have all nss-based application using p11-kit and reading certificates from the system-wide trust store!

…and this change needed to be turned off specifically for Firefox due to #126065 — the current form of p11-kit integration is not enough for Firefox, because the Mozilla code relies on the CKA_NSS_MOZILLA_CA_POLICY attribute, therefore the conversion of the builtin NSS certificate store to a format supported by p11-kit must be performed in a way that preserves that attribute (and some others too). If this is not fixed, at some future time (when the ESR branch will switch to 91) the firefox-esr and thunderbird packages would need the same treatment .

[rnhmjoj]

The solution should be to change pkgs.cacert and the security.pki module to generate the trust store in the p11-kit format and convert it to the openssl one. It shouldn't be too much work considering we can reuse the fedora script. I'll probably won't be able to work on this soon, though. Hopefully someone else will beat me to it.

[from-nibly]

Anything happening with this? I have a company cert that I need java to accept. If there is a workaround to that I can make a separate ticket.

[rnhmjoj]

@from-nibly See this https://discourse.nixos.org/t/custom-ssl-certificates-for-jdk

[uri-canva]

Was there ever any documentation produced about what the right pattern is? I see a lot of uses of ${cacert}/etc/ssl/certs/ca-bundle.crt, both within nixpkgs and other repos using nix, which if I understand correctly is not the right thing to do: https://sourcegraph.com/search?q=context%3Aglobal+%24%7Bcacert%7D%2Fetc%2Fssl%2Fcerts%2Fca-bundle.crt&patternType=standard&sm=1&groupBy=repo.

If I understand it correctly cacert is a package that should be used by NixOS, but not by packages themselves, packages should assume there is a ca store either defined in the NIX_SSL_CERT_FILE environment variable or in /etc/ssl/certs/ca-certificates.crt. I'm not sure what this means for macOS, where the system cert store is in the keychain?

[uri-canva]

One particular example is fetchgit setting GIT_SSL_CAINFO and requiring NIX_GIT_SSL_CAINFO to override it, instead of NIX_SSL_CERT_FILE (if I understand it correctly, there's a comment that mentions NIX_SSL_CERT_FILE but it implies both need to be set).

[vcunat]

Yes, it's still not at all unified even in the perspective of static vs. OS-provided. Apparently different packages/maintainers differ in opinion here.