Missing interpreters aren't skipped under pyenv

[florimondmanca]

EDIT: the solution is to activate all installed targeted Python versions in pyenv, see #252 (comment).

---

Describe the bug

The shims inserted by pyenv into the shell fool the py.path.local.sysfind() call made here, and results in Nox not correctly skipping missing interpreters.

How to reproduce

• Have pyenv installed with at least 2 different Python minor versions.
  • Docker setup (installing may take a while):

$ docker pull themattrix/pyenv
$ docker run --rm -it themattrix/pyenv bash
# pyenv install 3.6.5
# pyenv install 3.7.0
# pyenv global 3.7.0
# pip install nox

• Sample noxfile.py:

import nox


@nox.session(python=["3.6", "3.7"])
def test(session):
    pass

• Copy-paste, then run $ pbpaste > noxfile.py on macOS, or in the Docker container:

# apt update
# apt install nano
# touch noxfile.py
# nano noxfile.py
... Paste contents within nano ...

• Run $ nox.

Expected behavior

All sessions except the one using the Python version currently setup by pyenv (3.7 in my case) should be skipped, as explained in the docs

Actual behavior

Nox tries to run all versions, and fails. In my case, it fails on 3.6.

Full output:

$ nox
nox > Running session test-3.6
nox > Creating virtualenv using python3.6 in .nox/test-3-6
nox > Command /Users/florimond/.pyenv/versions/3.7.3/bin/python3.7 -m virtualenv /Users/florimond/Desktop/nox-pyenv/.nox/test-3-6 -p python3.6 failed with exit code 127:
pyenv: python3.6: command not found

The `python3.6' command exists in these Python versions:
  3.6.8

Running virtualenv with interpreter /Users/florimond/.pyenv/shims/python3.6
nox > Session test-3.6 failed.
nox > Running session test-3.7
nox > Creating virtualenv using python3.7 in .nox/test-3-7
nox > Session test-3.7 was successful.
nox > Ran multiple sessions:
nox > * test-3.6: failed
nox > * test-3.7: success

Extra debugging material

• Output of $ pyenv versions on my machine:

  system
  2.7.15
  3.5.6
  3.6.8
* 3.7.3 (set by /Users/florimond/.pyenv/version)

• Output of checking for Python versions on my machine:

$ which python3.6
/Users/florimond/.pyenv/shims/python3.6
$ which python3.7
/Users/florimond/.pyenv/shims/python3.7
$ python3.6 -V
pyenv: python3.6: command not found


The `python3.6' command exists in these Python versions:
  3.6.8

$ python3.7 -V
Python 3.7.3

• Output of $ nox -l on my machine:

$ nox -l
Sessions defined in /Users/florimond/Desktop/nox-pyenv/noxfile.py:

* test-3.6
* test-3.7

sessions marked with * are selected, sessions marked with - are skipped.

Possible solutions

There's a difference in output when running sysfind() on a version that's available but not selected, and the version that's currently selected. Example in my case (running 3.7 with 3.6.8 available):

>>> import py
>>> py.path.local.sysfind("python3.6")
local('/Users/florimond/.pyenv/shims/python3.6')
>>> py.path.local.sysfind("python3.7")
local('/Users/florimond/.pyenv/versions/3.7.3/bin/python3.7')
>>> 

We could detect that we're running under pyenv, and if so only use the result from sysfind() if it doesn't contain shims in the path? 🤷‍♂️

(We shouldn't rely on the output of sysfind to detect that we're using pyenv though, because the .pyenv shown there is the PYENV_ROOT, and that is configurable. I'd suggest to make a $ pyenv --version call instead.)

[florimondmanca]

Found the solution — I simply needed to activate all relevant Python versions in pyenv:

$  pyenv global 3.7.3 3.6.8

Then $ nox runs like ✨.

I don't think there's actually a bug in Nox at this point — if trying to use a Python version that's not installed by pyenv, e.g. 3.4 in my case, the version is skipped as expected.

So I'm going to close this. 😄 Hopefully this helps anyone stumbling upon this issue?

[theacodes]

@pszynk brought up that Nox's behavior with pyenv may not be correct:

Only local python interpreter versions should be considered when using pyenv local.
If we add python interpreter with pyenv global 3.6.12 (for example) the nox session is executed. Otherwise we got an error

@pszynk to make sure I understand your issue correctly, you want Nox to not use any of the global pyenv interpreters if local pyenv interpreters are configured?

[pszynk]

@theacodes first of all thanks for your contribution, this is a great project.

Sorry I didn't make myself clear and for the delayed response. I needed some time to investigate my issue.

My conclusions:

1. Why did I get a RuntimeError when running nox:

$ nox -s test -p 3.8

nox > Running session test-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/test-3-8
nox > Command /home/user/.pyenv/versions/3.9.1/envs/nox/bin/python3.9 -m virtualenv /home/user/tmp/test-nox/.nox/test-3-8 -p python3.8 failed with exit code 1:
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'
nox > Session test-3.8 failed.

This is due to my misuse of pyenv + pyenv-virtualenv (or a bug perhaps). One of pyenv managed virtualenvs had a symbolic link: python3.8 -> python but the python binary was in fact an interpreter version 3.9.1. How did it happen, I've no idea, and I cannot reproduce this. Still I don't think it really matters.

The point is that nox discovered an interpreter matching 3.8, while virtualenv did not, and this can be an issue.

2. Python interpreter discovery is inconsistent between nox and virtualenv:

Looking at both nox and virtualenv code I can see that an interpreter discovery can be tricky. Question is, if this logic has to be implemented by both packages? In case of the virtualenv this functionality is under development. They offer a plugin-based approach python-discovery, which could be an answer when you want to consider only local interpreters when using pyenv.

If nox should do interpreter discovery, could it possibly check if virtualenv found the proper interpreter. Unfortunately virtualenv throws only a generic RuntimeError on failure.

3. Solution with pyenv global <intepreter_1> <interpreter_2> ... is tricky!

Beware, virtualenv caches the results of interpreter discovery between executions (Linux: ~/.local/share/virtualenv). So what happens when nox searches for interpreters and there are pyenv installed ones present?

1. $ nox -s test -p 3.8
2. nox will find the executable like ~/.pyenv/shims/python3.8 thinking it is a python interpreter and pass -p python3.8 to virtualenv
3. virtualenv will again search for interpreter matching python3.8 and also find a path ~/.pyenv/shims/python3.8
4. if virtualenv finds path ~/.pyenv/shims/python3.8 is in cache -> goto 8
5. if path is not in the cache, virtualenv executes the file to gather information about the interpreter
6. if pyenv interpreter is not local or global shim wraper failes and virtualenv raises:

   RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'
   

7. if pyenv interperter is local or global shim executes proper python interpreter and virtualenv caches interpreter info, like the real path of the executable not the shim!.
8. virtualenv uses cached path of the real interpreter

That means that once the you make a pyenv interpreter local or global and create a virtual environment, and later remove it from local and global, virtualenv will still be able to find this interpreter looking into cache!

4. Should nox discover all pyenv installed interpreters?

The whole point of using pyenv locally is to create a specific environment for your project. So if I'm executing nox in my workspace directory, I'd expect it to stick to the environment I've prepared.

So yes, in my case the best solution would be to ignore all non-local pyenv interpreters, but...
If sb is using pyenv to install multiple interpreter versions in a docker container for testing purposes...
then I guess current solution is ok.

And If you want to execute only with a subset of interpreters with nox there is always an --pythons option

4. Summary

Is it possible to make nox and virtualenv interpreter discovery consistent?

The visibility of pyenv interpreters could be solved by virutalenv discovery plugins.

[theacodes]

I'm not sure I really understand how this is an issue? Nox looks at the path to discover pythons on unixes, and if it's on the path it's fair game. I don't see how that's inconsistent with virtualenv's behavior? Can you show me a case where Nox files pyenv's interpreter but virtualenv doesn't?

…

On Wed, Feb 10, 2021 at 2:24 PM pszynk ***@***.***> wrote: @theacodes <https://github.com/theacodes> first of all thanks for your contribution, this is a great project. Sorry I didn't make myself clear and for the delayed response. I needed some time to investigate my issue. My conclusions: 1. Why did I get a RuntimeError when running nox: $ nox -s test -p 3.8 nox > Running session test-3.8 nox > Creating virtual environment (virtualenv) using python3.8 in .nox/test-3-8 nox > Command /home/user/.pyenv/versions/3.9.1/envs/nox/bin/python3.9 -m virtualenv /home/user/tmp/test-nox/.nox/test-3-8 -p python3.8 failed with exit code 1: RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8' nox > Session test-3.8 failed. This is due to my misuse of pyenv + pyenv-virtualenv (or a bug perhaps). One of pyenv managed virtualenvs had a symbolic link: python3.8 -> python but the python binary was in fact an interpreter version 3.9.1. How did it happen, I've no idea, and I cannot reproduce this. Still I don't think it really matters. The point is that nox discovered an interpreter matching 3.8, while virtualenv did not, and this can be an issue. 2. Python interpreter discovery is inconsistent between nox and virtualenv : Looking at both nox and virtualenv code I can see that an interpreter discovery can be tricky. Question is, if this logic has to be implemented by both packages? In case of the virtualenv this functionality is under development. They offer a plugin-based approach python-discovery <https://virtualenv.pypa.io/en/20.4.2/extend.html#python-discovery>, which could be an answer when you want to consider only local interpreters when using pyenv. If nox should do interpreter discovery, could it possibly check if virtualenv found the proper interpreter. Unfortunately virtualenv throws only a generic RuntimeError on failure. 3. Solution with pyenv global <intepreter_1> <interpreter_2> ... is inaccurate In fact, there is no need to 'activate' pyenv environments, that is make them local or global. nox will discover all interpreters installed by pyenv regardless. aforementioned line: https://github.com/theacodes/nox/blob/d6a0ba5f8a8d34198ecd381faf9093ab77c3cb5e/nox/virtualenv.py#L354 should discover all shims in pyenv directory: ~/.pyenv/shims/ and shims for executables appear after command pyenv rehash 4. Should nox discover all pyenv installed interpreters? The whole point of using pyenv locally is to create a specific environment for your project. So if I'm executing nox in my workspace directory, I'd expect it to stick to the environment I've prepared. So yes, in my case the best solution would be to ignore all non-local pyenv interpreters, but... If sb is using pyenv to install multiple interpreter versions in a docker container for testing purposes... then I guess current solution is ok. And If you want to execute only with a subset of interpreters with nox there is always an --pythons option 4. Summary Is it possible to make nox and virtualenv interpreter discovery consistent? The visibility of pyenv interpreters could be solved by virutalenv discovery plugins. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#252 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5I45XYX7FTDKUO3Z7ZHTS6LMQHANCNFSM4I3N7ZFA> .

[pszynk]

The example would be the original issue.

When you install new pyenv interpreter and not make it local or global, nox thinks it has found the interpreter path, when in fact it's just found the shim. That's why the virtualenv logs the error and session fails:

RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'

I was mistaken about what really happens behind the scenes and I've fixed this in my previous comment.

But the point is that virtualenv has more sophisticated methods when it comes to interpreter discovery, and it would be nice if nox could make use of them. Still, should probably wait for a stable virtualenv API.

For now, nox could check the virtualenv stdout for this error and ignore the session instead of failing.
Not the most elegant of solutions, but I can make PR for that.

[theacodes]

I reproduced the described behavior, but I'm still a little unsure on a few points.

But the point is that virtualenv has more sophisticated methods when it comes to interpreter discovery, and it would be nice if nox could make use of them.

Does it, though? Nox invokes virtualenv directly to create the virtualenv with the given interpreter version & I don't see any difference between their behavior in this case:

$ virtualenv -p python3.7 env
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'

$ nox -s build
nox > Running session build
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/build
nox > Command ~/.pyenv/versions/3.9.1/bin/python3 -m virtualenv /Users/stargirl/workspace/blog.thea.codes/.nox/build -p python3.7 failed with exit code 1:
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'
nox > Session build failed.

Nox and Virtualenv are consistent here. So I'm not sure what we should do. We can try to detect pyenv stubs that are inactive and skip those environments, but I wanted to get some clarity on the assertion that virtualenv somehow does things differently that we do.

[pszynk]

What about this example:

virtualenv fails

$ virtualenv -p python_not_installed env
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python_not_installed'

nox skips

$ nox -s build -p python_not_installed
nox > Running session build
nox > Session build skipped: Python interpreter python_not_installed not found.

If virtualenv is used to create the virtual environment, why should nox be concerned with resolving the interpreter? virtualenv does it anyway. nox could leave this job to virtualenv and really would need to discover interpreter only when bare -m venv is to be used.
Skip the session if virutalenv failed to discover interpreter, fail otherwise.

But the point is that virtualenv has more sophisticated methods when it comes to interpreter discovery, and it would be nice if nox could make use of them.

Does it, though? Nox invokes virtualenv directly to create the virtualenv with the given interpreter version & I don't see any difference between their behavior in this case:

It has this system of plugins that allows for custom logic in interpreter discovery (option --discovery). This could be used let's say for selecting only local pyenv interpreters and ignoring the global ones.

[theacodes]

Virtualenv isn't our only backend, so that's one compelling reason why we do the lookup ourselves.

Skip the session if virutalenv failed to discover interpreter, fail otherwise.

We can modify Nox to detect pyenv stubs that aren't available and skip those sessions. That should resolve this issue for now.

This could be used let's say for selecting only local pyenv interpreters and ignoring the global ones.

Sure, it could be, but presently it is not.

[mforbes]

One slightly off-topic question: If an interpreter cannot be found, is there some way to specify a fallback? For example, how could one express the following to nox: If the requested python interpreter is not found, try to install it with conda, then re-run using that interpreter?

This can almost be done by having

# noxfile.py
import nox

@nox.session(python=["3.6", "3.7", "3.8"])
def test(session):
    session.install(".[test]")
    session.run("pytest", "-n", "4")

@nox.session(venv_backend="conda", python=["3.6", "3.7", "3.8"])
def test_conda(session):
    session.install(".[test]")
    session.run("pytest", "-n", "4")

but how can I have the test session delegate to the test_conda session when an interpreter is missing while also not running extranious tests in the case where the interpreters do exist.

I know that I can just use the "conda" backend all the time, but this can be significantly slower. What I am getting at is some way to use conda to get minimial enviroments fairly quickly to provide missing python interpreters, but still use the "Virtualenv" backend for the actual tests.

Sorry if this is noise and discussed somewhere, but I am still trying to wrap my head arround how this all works. Thanks!