avoid spurious warning about basepython setting for default factors

[freddrake]

Specifying a basepython setting for any environment including a pyXY factor (including the pyXY environments themselves) can trigger a spurious warning about the configured interpreter, even if the interpreter does provide the expected version.

The warning should only be generated if the specified interpreter does not conform to the expected version. Checking the return code from a command like this should do the trick:

{envdir}/bin/python -c 'import sys; sys.exit(sys.version_info[:2] != (X, Y))'

A return code of 0 indicates the interpreter specified conforms to the version indicated by the pyXY factor; 1 indicates a different version is provided, and anything else would be a real shock.

The the return code is 0, the indicated interpreter should be used regardless of the setting of the ignore_basepython_conflict setting, and no warning should be generated.

Previous discussion from the mailing list:
https://mail.python.org/mm3/archives/list/[email protected]/thread/CY2PB35PDX4IZAT73FRDCLSGQ4L2CPR4/

[freddrake]

A further point: the documentation needs to explain that the "default" Python version factors are about the interpreter version, not about the specific installation.

[obestwalter]

Hi @freddrake, thanks for reporting this.

I just wrote a dirty little script that grabs a bunch of tox.ini files from github and and runs parseconfig over them. Quite a few of them generated that warning.

On a first skim quite a few of them also don't look like a warning should be generated, so maybe that warning needs a bit of refinement?

... this is out of ~1800 config files, so not that many after all, but maybe worth looking into.

https://gist.github.com/obestwalter/ba3199c5564c35348abacd11db1c3d4d

Gist

Running tox 3.1 parseconfig over a random collection of tox.inis grabbed from github

Running tox 3.1 parseconfig over a random collection of tox.inis grabbed from github

[freddrake]

The warning is generated even when the interpreter configured via basepython is the right version; that can only be determined with the interpreter is present. When skip_missing_interpreters would cause an environment to be skipped, there shouldn't be a warning, since the version of the interpreter can't be determined. I'm fairly sure this is agreed, based on Bernát's message:

https://mail.python.org/mm3/archives/list/[email protected]/message/DX56THKBKNDACENE5BIZJGVGFFTAL2ZX/

Given this, the warning should never be generated for missing interpreters, and the version check needs to be performed for any interpreters present (including those for which PATH will be searched, even if the default basepython value is used).

I suspect that would make a significant difference in the outcome of your survey.

Mailman 3 [tox-dev] Re: Question about basepython setting - tox-dev - python.org

[obestwalter]

I suspect that would make a significant difference in the outcome of your survey.

I wasn't involved in the design and implementation of this, so can go about this all nice and naive :) From that naive perspective I would expect that those warnings are generated in a sensible way independent of the interpreters being installed - purely analysing the internal consistency of the configuration.

[obestwalter]

P.S. I added the scripts (grab and analyze) to the gist, if someone else wants to play with this and make sure this works o.k. in the wild. An access token is necessary in the same directory in token.txt for grab.py to work.

[freddrake]

I'm not sure what "consistency" would mean here.

The warning is not generated if the string value of basepython matches the default value for the factor. This is reasonable if you trust no one to have a pythonX.Y on their PATH that's really some other version. Arguably, that would be a pretty broken system.

Your tox-issue-908-check.txt doesn't indicate which of the reported conflicts you think are correct and which are not, so it's hard to comment on that.

My understanding of Bernát's comment on the mailing list is that the intent of the warning is to ensure that if the pyXY factor is applied, the interpreter used is actually an X.Y implementation. My understanding may be incorrect.

[obestwalter]

I was approaching this from a naive user perspective and wanted to see how many warnings this would generate if one runs it much like a naive user who checks out the code of some project and wants to start contributing. I ran the script on my devbox where I have 2.7, 3.4 and 3.6 installed, so not a completely uncommon setup.

My current feeling without having looked at the code in depth is, that while this warning moves towards fixing #477, we are not quite there yet. I think the plan is correct to warn for now and then throw an exception in 4.0 but I think we will need a bit more explanation/documentation for this.

[obestwalter]

Your tox-issue-908-check.txt doesn't indicate which of the reported conflicts you think are correct and which are not, so it's hard to comment on that.

Because I did not look at them in depth. I skimmed the inis with the warnings and a lot of them did not look to me like they would justify such a warning, so I posted it all here to motivate further analysis of this. I mainly had fun writing the scripts and ran them. I really am trying to keep the perspective of the normal user here. If we as experienced devs / contributors have a hard time figuring out what is going on, what chance does the normal user have? They will tend to ignore the warning, if they find no more explanations for this. I think we can improve on that.

[warsaw]

Landed here after searching for my problem. I'm using tox 3.2.1 and I can't figure out how to test one of my projects with a Python built from cpython's git repo. IIRC, I used to just set basepython = /path/to/my/built/python.exe (on macOS) and it would just work. Now I get a warning about the basepython conflict and ignore_basepython_conflicts=True doesn't help.

FWIW, I am trying to test importlib_resources with a feature branch of Python 3.8 using the py38 factor. Is this possible any more? It would be really sad if this feature were no longer available in tox.

[freddrake]

I've interpreted the response I received to this issue as "don't use a factor that looks like pyXY". Making this a special case seems really wrong to me, but then so does most of the current Python packaging & testing "best practice", so... leaves me wishing buildout hadn't ripped out support for multiple Python interpreters.

[gaborbernat]

Not at all; we are actually planning to implement a more accurate check (e.g. do not trust the executable name, but call out to Python to tell it). It's just our availability for this stopped this not being a reality yet. I'll try to tackle it next week.

[freddrake]

Bernát, thank you!

I'm glad you're planning to proceed with the more careful check. Getting the version from the specified interpreter seems much more useful. The behavior of ignoring the configured interpreter when ignore_basepython_conflicts=True was something I found truly mystifying.

I really appreciate the effort that's gone into tox, so I hope I don't come off as hostile. I know some people think I do.

[gaborbernat]

#983 should do it.