MacOS: Python shebang issue #123067

bergkvist · 2021-05-15T06:34:48Z

Describe the bug
On MacOS, nested shebangs/shebangs pointing at scripts are not allowed. When using pkgs.python39.withPackages(...) and installing packages with pip into a sandbox environment, binary executables get a shebang that makes them unable to execute.

To Reproduce

Create shell.nix, and run nix-shell:

let
  pkgs = import <nixpkgs> {};
  python = pkgs.python39.withPackages(ps: [ ps.pip ]);
in pkgs.mkShell {
  buildInputs = [ python ];
  shellHook = ''
    export PIP_DISABLE_PIP_VERSION_CHECK=1
    export PIP_PREFIX="$(pwd)/_build/pip"
    export PYTHONPATH="$PIP_PREFIX/lib/python3.9/site-packages:$PYTHONPATH"
    export PATH="$PIP_PREFIX/bin:$PATH"
  '';
}

Install ipython with pip, and try to execute it.

[nix-shell]$ pip install ipython
[nix-shell]$ ipython

/Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 3: import: command not found
/Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 4: import: command not found
from: can't read /var/mail/IPython
/Users/tobias//nix-python-issue/_build/pip/bin/ipython: line 7: syntax error near unexpected token `('
/Users/tobias//nix-python-issue/_build/pip/bin/ipython: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'

Expected behavior
It should be possible to do $ pip install ipython, followed by $ ipython - and have ipython actually execute successfully.

Observations

The shebang in _build/pip/bin/ipython points to a script, which is not allowed on MacOS (only Linux):

[nix-shell]$ cat $PIP_PREFIX/bin/ipython

#!/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

If /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9 was an executable, everything would be fine. But since this is a script, with its own shebang, this causes issues:

[nix-shell]$ cat /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9

#! /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env'
export NIX_PYTHONEXECUTABLE='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
export NIX_PYTHONPATH='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/lib/python3.9/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/7pjbbmnrch7frgyp7gz19ay0z1173c7y-python3-3.9.2/bin/python3.9"  "$@"

Metadata

system: "x86_64-darwin"
host os: Darwin 20.3.0, macOS 10.16
multi-user?: no
sandbox: no
version: nix-env (Nix) 2.3.10
channels(tobias): "darwin, nixpkgs-21.05pre284563.ab6943a7450"
nixpkgs: /Users/tobias/.nix-defexpr/channels/nixpkgs

The text was updated successfully, but these errors were encountered:

FRidh · 2021-05-15T10:02:06Z

Duplicate of #65351

bergkvist · 2021-05-15T15:39:16Z

Some more investegation:

[nix-shell]$ which pip

/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/pip

Looking at our pip executable (which is also a wrapped shell script):

[nix-shell]$ cat /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/pip

#! /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env'
export NIX_PYTHONEXECUTABLE='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
export NIX_PYTHONPATH='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/lib/python3.9/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/ccm6jcg1il8wdshiavrbc65p3s6i8rbl-python3.9-pip-21.0.1/bin/pip"  "$@"

What happens if we try to modify NIX_PYTHONEXECUTABLE?

# ...
export NIX_PYTHONEXECUTABLE='this-is-a-test'
# ...

... and then reinstall ipython:

[nix-shell]$ rm -rf _build && pip install -q ipython && cat _build/pip/bin/ipython

#!this-is-a-test
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

So it seems like NIX_PYTHONEXECUTABLE decides what the shebang will look like when pip generates executable python scripts.

bergkvist · 2021-05-16T04:03:28Z

I'm guessing a solution (on MacOS) here would be to set NIX_PYTHONEXECUTABLE to something like

# ...
export NIX_PYTHONEXECUTABLE='/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
# ...

But it seems like NIX_PYTHONEXECUTABLE changes behaviour when containing spaces:

[nix-shell]$ rm -rf _build && pip install -q ipython && cat _build/pip/bin/ipython

#!/bin/sh
'''exec' "/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9" "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

Which doesn't work

[nix-shell]$ ipython
/Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 2: /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9: No such file or directory
/Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 2: exec: /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9: cannot execute: No such file or directory

Manually modifying the ipython shebang (_build/pip/bin/ipython) to this makes it work:

#!/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

[nix-shell]$ ipython
Python 3.9.2 (default, Apr  8 2021, 19:41:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.23.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

bergkvist · 2021-05-16T05:13:43Z

Okay, through some experimentation, I discovered a hack for working around the issue with spaces in NIX_PYTHONEXECUTABLE.

# ...
export NIX_PYTHONEXECUTABLE='/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash" "/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
# ...

Notice the double quotes in the middle.

This causes the following shebang expression to be generated instead:

[nix-shell]$ rm -rf _build && pip install -q ipython && cat _build/pip/bin/ipython

#!/bin/sh
'''exec' "/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash" "/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9" "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

Which actually seems to work:

[nix-shell]$ ipython
Python 3.9.2 (default, Apr  8 2021, 19:41:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.23.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

bergkvist · 2021-05-16T05:25:21Z

Since this also works around all the shebang-limitations (like maximum length, number of arguments, pointing to a script etc), I imagine this should be fairly polymorphic with respect to different unix-platforms.

One would need to check whether the python interpreter is a binary file or a text file before determining what to use for NIX_PYTHONEXECUTABLE though.

The reference to /bin/sh could be a problem for reproducibility since it is outside of /nix/store/... (if the user has linked a non-POSIX compliant shell or something else here).

bergkvist · 2021-05-16T06:55:24Z

`NIX_PYTHONEXECUTABLE` was introduced in this PR: #65454

It works by setting sys.executable in Python.

nixpkgs/pkgs/development/interpreters/python/sitecustomize.py

Lines 31 to 35 in 3280de8

    
           executable = os.environ.pop('NIX_PYTHONEXECUTABLE', None) 
        
           prefix = os.environ.pop('NIX_PYTHONPREFIX', None) 
        
           if 'PYTHONEXECUTABLE' not in os.environ and executable is not None: 
        
               sys.executable = executable

pip uses `sys.executable` for deciding on what shebang to use when creating `_build/pip/bin/ipython`.

ScriptMaker._get_shebang()
https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/scripts.py#L164

get_executable()
https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/util.py#L312

This code puts double quotes around our executable if it contains spaces, and doesn't start with a double quote:
enquote_executable(executable) https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/scripts.py#L63

This code creates the exec-multiline shebang if it detects any spaces in the shebang (or we exceed the maximum shebang-length):
ScriptMaker._build_shebang():
https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/scripts.py#L147-L155

bergkvist · 2021-05-16T07:11:03Z

So turns out (from looking at pip source code) that we can put double quotes at the start/end of NIX_PYTHONEXECUTABLE, to make it slightly more explicit:

[nix-shell]$ cat $(which pip)

#! /nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env'
export NIX_PYTHONEXECUTABLE='"/nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash" "/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/bin/python3.9"'
export NIX_PYTHONPATH='/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/lib/python3.9/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/vr4yqqmx0s999xspgprnw7r3d1609hac-python3.9-pip-21.0.1/bin/pip"  "$@"

bergkvist · 2021-05-16T13:59:12Z

Counterexample to where the NIX_PYTHONEXECUTABLE from above doesn't work:

[nix-shell]$ pip install live-server

Collecting live-server
  Using cached live_server-0.9.9-py3-none-any.whl (5.9 kB)
Collecting beautifulsoup4==4.6.3
  Using cached beautifulsoup4-4.6.3-py3-none-any.whl (90 kB)
Collecting Click==7.0
  Using cached Click-7.0-py2.py3-none-any.whl (81 kB)
Collecting tornado==5.1.1
  Using cached tornado-5.1.1.tar.gz (516 kB)
    ERROR: Error [Errno 2] No such file or directory: '"/nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash" "/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/bin/python3.9"' while executing command python setup.py egg_info
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '"/nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash" "/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/bin/python3.9"'

It seems like sys.executable is expected to be a valid path, meaning we can't use the bash-hack. Although we get a working shebang - something else ends up failing as a result.

stale · 2021-11-16T12:22:24Z

I marked this as stale due to inactivity. → More info

NilsIrl · 2022-02-21T09:23:01Z

I'm having the same issue with ruby and the cewl package.

bergkvist · 2022-02-21T17:15:51Z

@NilsIrl I'm assuming you are using ruby.withPackages(...) to set up ruby if you are getting this issue. Now that makeBinaryWrapper has been merged into nixpkgs, you can try this out to see if it solves the problem:

# shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
  ruby = pkgs.ruby.override { makeWrapper = pkgs.makeBinaryWrapper; };
  rubyPkgs = ruby.withPackages(ps: []);
in
pkgs.mkShell {
  buildInputs = [ rubyPkgs ];
}

If it does, I can make a PR to make this the default wrapper for ruby

NilsIrl · 2022-02-21T18:45:46Z

That doesn't seem to be the case: https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/security/cewl/default.nix

bergkvist · 2022-02-21T18:48:11Z

@NilsIrl What nix expression are you using?

bergkvist · 2022-02-21T18:55:02Z

Oh, looking at pkgs.cewl I see the following:

nix-shell -p cewl
[nix-shell]$ head -n4 $(which cewl)

#!/nix/store/44mm7f7rsp5vc8dflkkn3pfbp8ghrjlf-wrapped-ruby-cewl-ruby-env/bin/ruby
#encoding: UTF-8

# == CeWL: Custom Word List Generator

[nix-shell]$ cat /nix/store/44mm7f7rsp5vc8dflkkn3pfbp8ghrjlf-wrapped-ruby-cewl-ruby-env/bin/ruby

#! /nix/store/a54wrar1jym1d8yvlijq0l2gghmy8szz-bash-5.1-p12/bin/bash -e
export BUNDLE_GEMFILE='/nix/store/s56719qzccbym8mbyfax6rsi2jx26zv3-gemfile-and-lockfile/Gemfile'
unset BUNDLE_PATH
export BUNDLE_FROZEN='1'
export GEM_HOME='/nix/store/yxapcqnbrfhgi7sl69hbqp18j0vs61sm-cewl-ruby-env/lib/ruby/gems/2.7.0'
export GEM_PATH='/nix/store/yxapcqnbrfhgi7sl69hbqp18j0vs61sm-cewl-ruby-env/lib/ruby/gems/2.7.0'
exec "/nix/store/ia70ss13m22znbl8khrf2hq72qmh5drr-ruby-2.7.5/bin/ruby"  "$@"

I'm on Linux right now though, but I can imagine that this wrapper looks the same on macOS as well - so we need to use makeBinaryWrapper for wrapped-ruby-cewl-ruby-env - since nested shebangs are not allowed on macOS.

bergkvist · 2022-02-21T19:11:29Z

@NilsIrl Kind of a "shotgun"-solution, but can you verify whether this works for you?

# shell.nix
let
  pkgs = import <nixpkgs> {
    overlays = [ (final: prev: { makeWrapper = prev.makeBinaryWrapper; }) ];
  };
in
pkgs.mkShell {
  buildInputs = [ pkgs.cewl ];
}

nix-shell shell.nix
[nix-shell]$ head -n4 $(which cewl)

#!/nix/store/d9shnlmjh51av5hsych4r4fpl0m9ydbf-wrapped-ruby-cewl-ruby-env/bin/ruby
#encoding: UTF-8

# == CeWL: Custom Word List Generator

Now, the shebang will be a binary wrapper instead of a shell script, which should make this work on macOS:

[nix-shell]$ cat /nix/store/d9shnlmjh51av5hsych4r4fpl0m9ydbf-wrapped-ruby-cewl-ruby-env/bin/ruby

...binary data...

# ------------------------------------------------------------------------------------
# The C-code for this binary wrapper has been generated using the following command:


makeCWrapper /nix/store/ia70ss13m22znbl8khrf2hq72qmh5drr-ruby-2.7.5/bin/ruby \
    --set 'BUNDLE_GEMFILE' '/nix/store/s56719qzccbym8mbyfax6rsi2jx26zv3-gemfile-and-lockfile/Gemfile' \
    --unset 'BUNDLE_PATH' \
    --set 'BUNDLE_FROZEN' '1' \
    --set 'GEM_HOME' '/nix/store/fal997gsyzyjrzcpx6cg2qirpyw1n0cn-cewl-ruby-env/lib/ruby/gems/2.7.0' \
    --set 'GEM_PATH' '/nix/store/fal997gsyzyjrzcpx6cg2qirpyw1n0cn-cewl-ruby-env/lib/ruby/gems/2.7.0'


# (Use `nix-shell -p makeBinaryWrapper` to get access to makeCWrapper in your shell)
# ------------------------------------------------------------------------------------

...binary-data...

NilsIrl · 2022-02-21T23:15:05Z

I can happily say that after waiting a few hours for stuff to build I have finally landed in a shell in which I can run cewl without issue.

bergkvist added the 0.kind: bug Something is broken label May 15, 2021

FRidh marked this as a duplicate of #65351 May 15, 2021

dotlambda added 6.topic: darwin Running or building packages on Darwin 6.topic: python labels May 15, 2021

bergkvist mentioned this issue May 24, 2021

Wrapper machinery should produce a tiny compiled binary #23018

Closed

bergkvist mentioned this issue Jun 4, 2021

Generate tiny compiled binary for wrapping executables #124556

Merged

9 tasks

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 16, 2021

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Feb 21, 2022

bergkvist mentioned this issue Feb 22, 2022

ruby-modules/bundler-env: Replace makeWrapper with makeBinaryWrapper to make bundled ruby apps work on macOS #161298

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MacOS: Python shebang issue #123067

MacOS: Python shebang issue #123067

bergkvist commented May 15, 2021

FRidh commented May 15, 2021

bergkvist commented May 15, 2021 •

edited

Loading

bergkvist commented May 16, 2021

bergkvist commented May 16, 2021

bergkvist commented May 16, 2021 •

edited

Loading

bergkvist commented May 16, 2021 •

edited

Loading

bergkvist commented May 16, 2021

bergkvist commented May 16, 2021

stale bot commented Nov 16, 2021

NilsIrl commented Feb 21, 2022

bergkvist commented Feb 21, 2022 •

edited

Loading

NilsIrl commented Feb 21, 2022

bergkvist commented Feb 21, 2022

bergkvist commented Feb 21, 2022 •

edited

Loading

bergkvist commented Feb 21, 2022 •

edited

Loading

NilsIrl commented Feb 21, 2022

MacOS: Python shebang issue #123067

MacOS: Python shebang issue #123067

Comments

bergkvist commented May 15, 2021

Observations

FRidh commented May 15, 2021

bergkvist commented May 15, 2021 • edited Loading

bergkvist commented May 16, 2021

bergkvist commented May 16, 2021

bergkvist commented May 16, 2021 • edited Loading

bergkvist commented May 16, 2021 • edited Loading

NIX_PYTHONEXECUTABLE was introduced in this PR: #65454

pip uses sys.executable for deciding on what shebang to use when creating _build/pip/bin/ipython.

bergkvist commented May 16, 2021

bergkvist commented May 16, 2021

stale bot commented Nov 16, 2021

NilsIrl commented Feb 21, 2022

bergkvist commented Feb 21, 2022 • edited Loading

NilsIrl commented Feb 21, 2022

bergkvist commented Feb 21, 2022

bergkvist commented Feb 21, 2022 • edited Loading

bergkvist commented Feb 21, 2022 • edited Loading

NilsIrl commented Feb 21, 2022

bergkvist commented May 15, 2021 •

edited

Loading

bergkvist commented May 16, 2021 •

edited

Loading

bergkvist commented May 16, 2021 •

edited

Loading

`NIX_PYTHONEXECUTABLE` was introduced in this PR: #65454

pip uses `sys.executable` for deciding on what shebang to use when creating `_build/pip/bin/ipython`.

bergkvist commented Feb 21, 2022 •

edited

Loading

bergkvist commented Feb 21, 2022 •

edited

Loading

bergkvist commented Feb 21, 2022 •

edited

Loading