nix-build uses fails on SSL peer certificate error where other nix commands do not, likely not respecting NIX_SSL_CERT_FILE

[alexcardell]

Describe the bug

Company has added ZScaler vpn, and so my nix-darwin home-manager set up began to fail with SSL peer certificate or SSH remote key was not OK when calling cache.nixos.org. There is a Zscaler Root CA in my keychain.

After a lot of tinkering with NIX_SSL_CERT_FILE and no luck, I decided a reinstall was the way as in the manual 3.6 example has it set before the installer runs.

After much more tinkering I have a working install: exporting a .p12 of all the CAs in the keychain, converting that to a .crt, let's call it zscaler-ca.crt and export NIX_SSL_CERT_FILE=zscaler-ca.crt; sh <(curl -L https://nixos.org/nix/install)

launchctl setenv NIX_SSL_CERT_FILE ...; launchctl kickstart -k system/org.nixos.nix-daemon did not help me, so I've also edited /Library/LaunchDaemons/org.nixos.nix-daemon.plist to set NIX_SSL_CERT_FILE on boot to zscaler-ca.crt (Interestingly the nix installer respects the NIX_SSL_CERT_FILE setting but it will still be the default value in the generated launch daemon configuration)

After the install the following all work:

• nix-env -iA ripgrep and any other packages works fine
• nix-channel --update nixpkgs
• nix-env -iA git and then git clone from github over https
• CURL_CA_BUNDLE=zscaler-ca.crt curl -vL https://github.com/LnL7/nix-darwin/archive/master.tar.gz -o out

But nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer, as per the nix-darwin installation docs, does not work, with the following error

% nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer              
warning: error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 324 ms
warning: error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 547 ms
warning: error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 1060 ms
warning: error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 2350 ms
error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': SSL peer certificate or SSH remote key was not OK (60)

This is the same error that kicked all this off. The same applies for nix-build https://github.com/NixOS/nixpkgs/archive/master.tar.gz -A hello

This is with CURL_CA_BUNDLE SSL_CERT_FILE and NIX_SSL_CERT_FILE environment variables all set to zscaler-ca.crt, and the nix-daemon plist setting NIX_SSL_CERT_FILE

Expected behavior

nix-build https://github.com/NixOS/nixpkgs/archive/master.tar.gz -A hello should pass, respecting NIX_SSL_CERT_FILE

nix-env --version output

nix-env (Nix) 2.13.2

Additional context

• macOS Montery 12.6
• M1 chip

Priorities

Add 👍 to issues you find important.

[alexcardell]

Setting CURL_CA_BUNDLE to the .crt in the nix-daemon.plist is also hnot helping

[alexcardell]

So this is solved on my end through the actions of the company admins rolling back the VPN so I could complete the nix-build phase for nix-darwin, and adding some exclusion for cache.nixos.org

The environment variable behaviour was inconsistent but I'm closing as I now can no longer reproduce

[rmcgibbo]

I also had this issue, and found that it only happens when I'm on the corporate vpn. When I disconnect, the error goes away. Thanks for your prior post on that, @alexcardell, otherwise I wouldn't have guessed it.

[DonPavlov]

I would not describe this as closed, as the issue is still persisting and only circumvented by not using the company network.

[lytedev]

Seconded. We'd love to adopt nix at my work but we need the ability to integrate the Zscaler certificate with Nix and it doesn't seem to work!

[aagatequinix]

this is definitely not fixed (on macos) and after consuming most of what Google and github has indexed on this topic, I'm still utterly confused why nix isn't using the local keychain successfully as other software does

[thufschmitt]

Just tried that on a Mac machine, it seems that NIX_SSL_CERT_FILE and SSL_CERT_FILE are picked-up at least partially:

$ NIX_SSL_CERT_FILE=/dev/null nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer
error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': Problem with the SSL CA cert (path? access rights?) (77)
$ SSL_CERT_FILE=/dev/null nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer
error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': Problem with the SSL CA cert (path? access rights?) (77)
$ nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer
# Works

Can you try with more verbosity, see what curl says about the CA file used? Something like nix-prefetch-url https://github.com/LnL7/nix-darwin/archive/master.tar.gz -vvvvv?

[lytedev]

FWIW, we ended up getting things working by dumping the cert/ca/chain/whatever and setting up nix to use it. We used the Determinate Systems installer as well. Here's the relevant part of the script:

# dump macOS cert bundle
sudo security export -t certs -p -o '/opt/zscaler.crt'

# ensure the zscaler cert bundle is world readable
sudo chmod 755 '/opt/zscaler.crt'

# install nix using the zscaler cert bundle
curl --proto '=https' --tlsv1.2 -sSf -L 'https://install.determinate.systems/nix' \
  | sh -s -- install --no-confirm --ssl-cert-file '/opt/zscaler.crt'

# merge Nix and zscaler cert bundles
# TODO: I expect this will break if/when nix updates their ca bundle?
cat '/opt/zscaler.crt' '/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt' \
  | sudo tee '/opt/nix-and-zscaler.crt'
sudo chmod 755 '/opt/nix-and-zscaler.crt'

# reconfigure nix to use our combined cert bundle instead
sudo sed -i '' '/^ssl-cert-file /d' '/etc/nix/nix.conf'
printf "\nssl-cert-file = /opt/nix-and-zscaler.crt\n" | sudo tee -a '/etc/nix/nix.conf'

# make the current user a trusted user
printf "\ntrusted-users = root %s\n" "$USER" | sudo tee -a '/etc/nix/nix.conf'

# source the nix integration script
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'

I'm not sure if this helps anybody, but this definitely got nix in the door at work with Zscaler for us.

[DonPavlov]

well that sounds like a lot of work, but doesn't fix the underlying issue of nix not handling the cert per default. But maybe a viable workaround. Due to corporate changes for me I am unable to try this out again. But will just try using it normally on a dedicated machine.

[Leandros]

Agreed. This is definitely not closed and easily reproducible on a machine using Zscaler.

[kagehisa]

Same problem here, setting NIX_SSL_CERT_FILE does work for some nix related commands like flake update but nix run for example fails due to SSL errors.

[malik-n]

Same issue here, using NixOS as WSL distro. nixos-rebuild works as root with no problem by setting the NIX_SSL_CERT_FILE variable to the zscaler certificate file. But this method does not work when using nix as normal user.

[janpawlowskiof]

I know nothing about certificates, but in my case $NIX_SSL_CERT_FILE was set to some custom .pem file, but i was getting an SSL error.

I removed (and backed up) the /etc/ssl/certs/ca-certificates.crt and created a symlink at /etc/ssl/certs/ca-certificates.crt to the file pointed by my $NIX_SSL_CERT_FILE.

Now it works!

I assume that really means nix does not always use the value of $NIX_SSL_CERT_FILE.
The downside is that it requires sudo, and I just pray that it does not break anything else in the system.

[wardnath]

FWIW, we ended up getting things working by dumping the cert/ca/chain/whatever and setting up nix to use it. We used the Determinate Systems installer as well. Here's the relevant part of the script:

# dump macOS cert bundle
sudo security export -t certs -p -o '/opt/zscaler.crt'

# ensure the zscaler cert bundle is world readable
sudo chmod 755 '/opt/zscaler.crt'

# install nix using the zscaler cert bundle
curl --proto '=https' --tlsv1.2 -sSf -L 'https://install.determinate.systems/nix' \
  | sh -s -- install --no-confirm --ssl-cert-file '/opt/zscaler.crt'

# merge Nix and zscaler cert bundles
# TODO: I expect this will break if/when nix updates their ca bundle?
cat '/opt/zscaler.crt' '/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt' \
  | sudo tee '/opt/nix-and-zscaler.crt'
sudo chmod 755 '/opt/nix-and-zscaler.crt'

# reconfigure nix to use our combined cert bundle instead
sudo sed -i '' '/^ssl-cert-file /d' '/etc/nix/nix.conf'
printf "\nssl-cert-file = /opt/nix-and-zscaler.crt\n" | sudo tee -a '/etc/nix/nix.conf'

# make the current user a trusted user
printf "\ntrusted-users = root %s\n" "$USER" | sudo tee -a '/etc/nix/nix.conf'

# source the nix integration script
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'

I'm not sure if this helps anybody, but this definitely got nix in the door at work with Zscaler for us.

This worked as well for me to get nix working with MacOS on my company Zscaler VPN.

[malik-n]

I assume that really means nix does not always use the value of $NIX_SSL_CERT_FILE. The downside is that it requires sudo, and I just pray that it does not break anything else in the system.

I also had the issue that putting the certificate in in /etc/ssl/certs and setting the environment variable only let me use the nix command with sudo.
I fixed it with setting:

nix.settings.ssl-cert-file = "/etc/ssl/certs/zscaler.crt";
security.pki.certificates = [
  "/etc/ssl/certs/zscaler.crt"
];

Sometimes I still get ssl errors, but then I redo the command untill the packages come through.