Exception from truststore when installing editable pip using not editable pip

[notatallshaw]

Description

This is on pip main, I think after #11647 has landed, when you install pip in editable mode using pip in not editable mode you get an exception, although the installation itself appears to succeed.

Expected behavior

No exception on editable install

pip version

24.2

Python version

3.12

OS

Linux

How to Reproduce

1. git clone https://github.com/pypa/pip
2. cd pip
3. Create and activate Python 3.12 virtual environment
4. python -m pip install .
5. python -m pip install -e .

Output

$ python -m pip install -e .
Obtaining file:///home/damian/opensource/support/pip/pip
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pip
  Building editable for pip (pyproject.toml) ... done
  Created wheel for pip: filename=pip-24.2.dev0-0.editable-py3-none-any.whl size=9633 sha256=ee68c183a99b2e54100600272d5340fcf701e747379917d6e609123361bb5615
  Stored in directory: /tmp/pip-ephem-wheel-cache-5tlhi9f3/wheels/74/89/b1/b2dda733bdd1d7513ad1e2f926bd41e01be53b3862742ca12b
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.2.dev0
    Uninstalling pip-24.2.dev0:
      Successfully uninstalled pip-24.2.dev0
Successfully installed pip-24.2.dev0
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/__main__.py", line 24, in <module>
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/_internal/cli/main.py", line 80, in main
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 101, in main
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 236, in _main
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/_internal/cli/index_command.py", line 160, in handle_pip_version_check
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/_internal/cli/index_command.py", line 95, in _build_session
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/_internal/cli/index_command.py", line 46, in _create_truststore_ssl_context
  File "/home/damian/opensource/support/pip/pip/.venv/lib/python3.12/site-packages/pip/_vendor/truststore/_api.py", line 135, in load_verify_locations
FileNotFoundError: [Errno 2] No such file or directory

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[ichard26]

Damian and I investigated this issue on the PyPA Discord. Here's a Discord link to the start of the conversation, but I will provide a summary of the important bits here as those details belong here anyway :)

The root issue is that the pip self version check uses its own requests session¹.

pip/src/pip/_internal/cli/index_command.py

Lines 159 to 166 in 8eadcab

	# Otherwise, check if we're using the latest version of pip available.
	session = self._build_session(
	options,
	retries=0,
	timeout=min(5, options.timeout),
	)
	with session:
	_pip_self_version_check(session, options)

Within the initialization of a new session, the truststore context is augmented with the certifi CA bundle.

pip/src/pip/_internal/cli/index_command.py

Lines 45 to 47 in 8eadcab

	ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
	ctx.load_verify_locations(certifi.where())
	return ctx

It's this line, specifically the certifi.where() call, that's causing this error. If I log the path returned by certifi.where() in this function while following the reproduction steps in this issue, the issue becomes clear:

There are two logs because there are two sessions being initialized², one for the install command and the other for pip's self version check. certifi.where() returns the same path for both calls which ends poorly when transitioning from a normal installation of pip to an editable installation. In the editable install, the old site-packages pip install (and consequently, the old certifi CA bundle) is deleted, breaking future sessions.

This can be inferred from the details, but I want to make it clear that this only affects the installation of pip itself. So, this isn't strictly a blocker for 24.2 as previously thought. Also, the resulting installation is unaffected as the self version check runs at the very end of the pip invocation, after the command terminates. However, I'll note that the exception discarding we do during the version check is useless as the session is created and errors out before the try... except block.

I haven't investigated potential solutions yet (it's late and I should go to bed soon), but I'd imagine one of the simpler patches would be to change the self version check to reuse the same PipSession initialized at start-up by the (network-requiring) command. This shouldn't result in additional expensive sessions being created since after #12637, only command invocations that require the network anyway will create a session and importantly run the self version check ³.

This issue also raises a larger question of whether we want to one day move pip off of certifi entirely. According to @notatallshaw, previously the truststore feature in 24.1 disabled certifi outright. Thus, developers and organisations who wanted to avoid using certifi could've used --use-feature=truststore to enforce the sole usage of system certificates. We doubt that there is anyone relying on this today, and more broadly this should be its own conversation, but this issue does bring it up as #11647 gets pip in the business of directly using certifi, instead of letting requests handle certifi usage for us.

Footnotes

1. We actually subclass requests.Session, but *hand waves* this isn't relevant here. ↩

2. This is wrong as the pip subprocesses involved in PEP 517 build requirements provisioning will create their own sessions. However, they shouldn't be affected as they shouldn't be installing pip in a way that changes the resources path dangerously (and if they do, frankly who cares...?). Anyhow, Disable self version check in PEP 517 pip subprocesses #12683 disabled the self version check in these subprocesses. ↩

3. Don't quote me on this, though. There could very well be a devil in the details that I'm missing. ↩

[notatallshaw]

Fixed by #12865