setuptools 50 fails with KeyError: 'distutils' when installing some packages

[enkidulan]

I got the following error when tried to install doit.

$ pip install doit
Collecting doit
  Using cached doit-0.32.0.tar.gz (1.4 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/enki/projects/.../.venv/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-t_g4mfyi/doit/setup.py'"'"'; __file__='"'"'/tmp/pip-install-t_g4mfyi/doit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-shhwaq7f
         cwd: /tmp/pip-install-t_g4mfyi/doit/
    Complete output (17 lines):
    /home/enki/projects/.../.venv/lib/python3.6/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
      warnings.warn("Setuptools is replacing distutils.")
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-t_g4mfyi/doit/setup.py", line 88, in <module>
        'doit = doit.__main__:main'
      File "/home/enki/projects/.../.venv/lib/python3.6/site-packages/setuptools/__init__.py", line 152, in setup
        _install_setup_requires(attrs)
      File "/home/enki/projects/.../.venv/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in _install_setup_requires
        dist.parse_config_files(ignore_option_errors=True)
      File "/home/enki/projects/.../.venv/lib/python3.6/site-packages/setuptools/dist.py", line 665, in parse_config_files
        self._parse_config_files(filenames=filenames)
      File "/home/enki/projects/.../.venv/lib/python3.6/site-packages/setuptools/dist.py", line 572, in _parse_config_files
        filenames = self.find_config_files()
      File "/home/enki/projects/.../.venv/lib/python3.6/site-packages/setuptools/_distutils/dist.py", line 353, in find_config_files
        sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
    KeyError: 'distutils'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[asfaltboy]

Could this be a side-effect of overriding distutils on stdlib the as described in #2350? I'll try to confirm the workaround (using SETUPTOOLS_USE_DISTUTILS=1 works and report back).

EDIT: the workaround worked fine, as long as I run it before installing python via apt-get / get-pip installations .

[enkidulan]

not likely, it rather seems that distutils is not in import "cache". I think that importing distutils at the top of setuptools/_distutils/dist.py should fix the issue, but I'm not sure if it's a good solution.

[Kwpolska]

The doit issue is caused by an import pip (to ensure it’s in a new enough version) in its setup.py (see pydoit/doit#369).

[enkidulan]

I'm not sure that it's doit issue. is import pip in setup.py is not allowed anymore? if so, why?

[jaraco]

I'll try to confirm the workaround (using SETUPTOOLS_USE_DISTUTILS=1 works and report back).

FYI, The two supported values for SETUPTOOLS_USE_DISTUTILS are stdlib (to opt out of the local distutils) and local (to opt into the local distutils).

[jaraco]

Aha - so I have a hint of what's going on.

In this method of the distutils hack, Setuptools aims to get out of the way, disabling the hack and falling back to stdlib imports of distutils when pip is running. pypa/pip#8761 is mentioned as the rational for this behavior. This method is assuming, as pip requires, that pip not be used as a library, but only be invoked through its command-line interface. The fact that you've imported setuptools, then imported pip, means you've violated this expectation and triggered the condition where although setuptools._distutils has been imported as sys.modules['distutils'], it's since been removed.

Perhaps the thing to do here is to raise an error in spec_for_pip if 'setuptools' in sys.modules. That approach wouldn't fix this error, but it might make the cause clearer.

Alternately, the routine could avoid clearing distutils if setuptools is already imported. That approach would possibly fix the behavior, but only if the undesirable pip behavior that's being avoided isn't triggered following the user's import of pip.

Should Setuptools support importing of pip after importing of setuptools? My instinct is no and to declare this failure as an unsupported use-case.

is import pip in setup.py is not allowed anymore? if so, why?

import pip has never been supported, and this failure is one reason why - maintaining interpreter-level compatibility across these tools and supported platforms is extremely difficult and relying on the hard interpreter boundary for pip invocation provides guarantees that enable the PyPA team to do hacky things like replace import distutils in a selective way.

Can you make a strong case why import pip is an essential behavior that should be supported?

[enkidulan]

I see, thanks for the explanation. Closing this issue as irrelevant then.

[xkszltl]

@jaraco

import pip has never been supported, and this failure is one reason why - maintaining interpreter-level compatibility across these tools and supported platforms is extremely difficult and relying on the hard interpreter boundary for pip invocation provides guarantees that enable the PyPA team to do hacky things like replace import distutils in a selective way.

Is import pip not supported in setup.py or everywhere?
I've seen people using it a lot to either check version, log package info, and install dependencies at the beginning of their script to make it convenient for end users.

[xkszltl]

And if it's not supported, should it simply fail (maybe with some environment detection) instead of asking why people uses an unsupported code later?

[jaraco]

According to using pip in your program, import pip is not supported anywhere.

#2381 takes some steps to avoid breaking under the conditions where pip is imported.

And if it's not supported, should it simply fail (maybe with some environment detection) instead of asking why people uses an unsupported code later?

Perhaps that's a question for the pip maintainers. My instinct is that users should be allowed to "import pip" at their own risk, so it shouldn't be strictly enforced.