On a relocatable Python install, setuptools should compute the C include path correctly

[njsmith]

You can distribute CPython as a "relocatable" zip file, meaning, you just drop the zip file where-ever you want, and it works. For example, on Linux:

$ wget https://pybi.vorpus.org/cpython_unofficial-3.11.0-1-manylinux_2_17_x86_64.pybi
$ mkdir cpython-3.11
$ cd cpython-3.11
$ unzip ../cpython_unofficial-3.11.0-1-manylinux_2_17_x86_64.pybi
$ bin/python -c 'print("Hello world")'
Hello world

(You can also try this on macos or windows if you prefer.)

The resulting python knows where it is, where wheels should be installed, and all that; see sysconfig.get_paths():

$ bin/python -c 'import sysconfig, pprint; pprint.pprint(sysconfig.get_paths())'
{'data': '/tmp/cpython-3.11',
 'include': '/tmp/cpython-3.11/include/python3.11',
 'platinclude': '/tmp/cpython-3.11/include/python3.11',
 'platlib': '/tmp/cpython-3.11/lib/python3.11/site-packages',
 'platstdlib': '/tmp/cpython-3.11/lib/python3.11',
 'purelib': '/tmp/cpython-3.11/lib/python3.11/site-packages',
 'scripts': '/tmp/cpython-3.11/bin',
 'stdlib': '/tmp/cpython-3.11/lib/python3.11'}

Almost everything works great. But! When you use setuptools to compile a C extension, it uses the wrong include path:

$ bin/python -m ensurepip
$ bin/python -m pip install -U pip setuptools
$ mkdir foo
$ cd foo
$ echo '#include <Python.h>' > foo.c
$ cat <<EOF >setup.py
from setuptools import setup, Extension
setup(name="foo", ext_modules=[Extension("foo", sources=["foo.c"])])
EOF
$ ../bin/python setup.py build_ext
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-x86_64-cpython-311
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/pyinstall/include/python3.11 -c foo.c -o build/temp.linux-x86_64-cpython-311/foo.o
foo.c:1:10: fatal error: Python.h: No such file or directory
    1 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

Notice that the gcc command line has -I/pyinstall/include/python3.11, which is a path on the machine where this Python interpreter was built, not the path where it's installed, or the path returned by sysconfig.get_paths().

I wasn't able to entirely follow where setuptools gets its include path from, but I think it's ultimately coming from distutils.sysconfig.get_python_inc, which invokes a number of strategies to find the include directory, but they're all based on static compile-time data, rather than the actual runtime location of the Python interpreter.

This is the only problem I've managed to find so far with these relocatable interpreters, so it would be nice if we could fix setuptools to use the correct include path :-).

[edmorley]

Hi! This seems like it might be a duplicate of #3657, which is now fixed in setuptools 67.2.0:
#3657 (comment)

Does newer setuptools resolve the issue for you? :-)

[lordmauve]

I have the same issue and it is not fixed in setuptools 67.2.0 or later. However I can confirm that it works with 65.1.1 and is broken in 65.2.0.

[lordmauve]

I believe that it is 9079e77 that breaks this. spec_prefix is None in my case.

[abravalheri]

Hi @lordmauve , since you are already investigating, would you like to provide a PR to the pypa/distutils project on github?

[lordmauve]

I agree that this does look like a duplicate of #3657 and #3778, and pypa/distutils#178. I don't know why I'm still seeing this issue though if all of those are closed.

[zjp]

In my case I was seeing it because build-time locations were in sysconfigdata. When we build ChimeraX we now set CONFINCLUDEPY, CONFINCLUDEDIR, INCLUDEPY, and INCLUDEDIR to the empty string.