Recommend pip instead of python -m pip, fix minor error

[mbargull]

There should be no advantage in using python -m pip over pip, so we should use the latter. Even under Windows we always have a pip.exe available, so really no reason not to use it.

Additionally, the docs say a pip install --no-deps --ignore-installed . invocation would not create egg-info dirs, which is not true, so I removed that part of the sentence.

[mbargull]

"which do not interact well with conda" should be applicable to both, egg-info dirs and pip-installed deps, so I left this in. @nicoddemus, would you agree this is fine?

[nicoddemus]

Sure, sounds fine to me! 👍

[ocefpaf]

See https://bugs.python.org/issue22295 and a more concise version at https://stackoverflow.com/questions/25749621/whats-the-difference-between-pip-install-and-python-m-pip-install for the reasons conda-forge chose python -m pip. Calling pip locally is fine as long as you know what you are doing, for the recipes we prefer to unambiguous and generic.

[mbargull]

Sorry, but I don't think this is really applicable here. That https://bugs.python.org/ issue and Stack Overflow question are about non-Conda installations.
python -m pip is in no way more less ambiguous than pip in a Conda environment.
Please reopen and let's discuss. cc @conda-forge/core.
[EDIT: s/more/less/ ...]

[mbargull]

The egg-info documentation fix applies either way.

[mbargull]

One could even argue -- by an, admittedly, long stretch -- that python -m pip is an a bit inferior solution, see, e.g., https://bugs.python.org/issue33053.

[ocefpaf]

Sorry, but I don't think this is really applicable here.

conda-forge provides also recipes, not only packages, the safest generic setup is with python -m.

That https://bugs.python.org/ issue and Stack Overflow question are about non-Conda installations.
python -m pip is in no way more ambiguous than pip in a Conda environment.

One can have python from conda, that comes without pip, and pip call pip from somewhere else. That is why we have python -m. If we wanted to the pedantic about it we should call $PYTHON -m or %PYTHON% -m, but we expected the user to always be in a conda python env.

Please reopen and let's discuss. cc @conda-forge/core.

We can discuss a closed issue and I'm happy to merge this if you can proof me that we can be safe with just pip in all setup where conda-forge recipes are used. I maintain many systems that uses conda-forge recipes and I can assure you that python -m saves me a lot of trouble.

One could even argue -- by an, admittedly, long stretch -- that python -m pip is an a bit inferior solution, see, e.g., https://bugs.python.org/issue33053.

Indeed a long stretch.

[djsutherland]

One can have python from conda, that comes without pip, and pip call pip from somewhere else. That is why we have python -m. If we wanted to the pedantic about it we should call $PYTHON -m or %PYTHON% -m, but we expected the user to always be in a conda python env.

This is a good point: if you forget to include the pip build requirement, just calling pip would call the system-provided pip, no?

[ocefpaf]

This is a good point: if you forget to include the pip build requirement, just calling pip would call the system-provided pip, no?

Yep. That is one of the reasons for python -m.

[jakirkham]

FWIW I can see value in cutting down the length of this command. It feels like a lot of cognitive load to be carrying around. Given we already tell people to add pip as a build requirement explicitly, I think the paranoia of getting the wrong pip is unjustified. If we really are this worried, we really should be using $PYTHON/%PYTHON% because we could pick up another python install, but we don't that. So why python -m pip?

On .egg-info, it isn't included actually. That said, pip seems to produce and include .dist-info, which is just as bad AFAIAC. It would be great if pip didn't do that or we could find an option to address that somehow. Have looked before, but didn't find anything. Suggestions welcome.

[mbargull]

It's just like calling another python if you forgot that dependency.

So, your arguments can be summarized with

there is a higher probability that one misses a dependency on pip than missing one on python

right? (With which I agree, of course.)
If the one or the other really saved or caused, resp., you "a lot of trouble" wouldn't it be better to have the linter check for pip being in requirments: build if pip install/python -m pip install is used? (Which is completely independent from this PR.)

Regarding $PYTHON, IMHO, rather depend on $PREFIX so we could deprecate those old language-specific env vars some time...

if you can proof me that we can be safe with just pip in all setup where conda-forge recipes are used

(Rhetoric question) How could I? I don't know "all setups" and there is a plethora of ways to mess with the execution of pip or python, so that's hardly provable. If you want to make it a little more robust that pip or python -m pip, you'd have to use "${PREFIX}/bin/pip" / "%PREFIX%\Scripts\pip".

[OT: Re: open/closed issue, that's just because of the "awareness factor" 😉 (hence the ping).]

[ocefpaf]

FWIW I can see value in cutting down the length of this command. It feels like a lot of cognitive load to be carrying around.

There is no cognitive load b/c people either copy-n-paste that. I would argue that should be in conda-skeleton to reduce it even further.

Given we already tell people to add pip as a build requirement explicitly, I think the paranoia of getting the wrong pip is unjustified.

Not a paranoia. A Fact. Again, conda-forge recipes are used outside of conda-forge and good practices do not hurt anyone.

[mbargull]

Re: egg-info:

> docker run --rm --entrypoint '' -it condaforge/linux-anvil /bin/bash                                                                                                                                                                                                     
[root@ea420dcab316 /]# PS1='$ '
$ mkdir -p /test/pkg/
$ cd /test
$ touch pkg/__init__.py
$ cat > setup.py <<'EOF'
> from setuptools import find_packages, setup
> setup(name='pkg', version='0.1', packages=find_packages())
> EOF
$ mkdir /out
$ . /opt/conda/bin/activate 
(root) $ pip install -t /out --no-deps --ignore-installed .
Processing /test
Installing collected packages: pkg
  Running setup.py install for pkg ... done
Successfully installed pkg-0.1
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(root) $ ls -lAR /out/
/out/:
total 0
drwxr-xr-x 1 root root  44 Apr  3 14:34 pkg
drwxr-xr-x 1 root root 142 Apr  3 14:34 pkg-0.1-py3.6.egg-info

/out/pkg:
total 0
-rw-r--r-- 1 root root  0 Apr  3 14:31 __init__.py
drwxr-xr-x 1 root root 46 Apr  3 14:34 __pycache__

/out/pkg/__pycache__:
total 0
-rw-r--r-- 1 root root 126 Apr  3 14:34 __init__.cpython-36.pyc

/out/pkg-0.1-py3.6.egg-info:
total 0
-rw-r--r-- 1 root root   1 Apr  3 14:34 dependency_links.txt
-rw-r--r-- 1 root root 118 Apr  3 14:34 installed-files.txt
-rw-r--r-- 1 root root 175 Apr  3 14:34 PKG-INFO
-rw-r--r-- 1 root root 123 Apr  3 14:34 SOURCES.txt
-rw-r--r-- 1 root root   4 Apr  3 14:34 top_level.txt

[djsutherland]

I've definitely forgotten to include pip in recipes before (when changing from setuptools to pip), and it fails quickly this way. If we switched to just pip, it'd just produce an empty package, which is definitely far less noticeable. A linter check would help with that, though, and we should probably add some kind of check for empty packages as well.

[mbargull]

conda-build >=3.6 already has a check for empty packages https://github.com/conda/conda-build/blob/3.8.0/CHANGELOG.txt#L131-L133 😄

[jjhelmus]

That said, pip seems to produce and include .dist-info, which is just as bad AFAIAC.

The presence of the .dist-info or even the .egg-info directories are VERY useful when trying to get pip and conda to play nicely together. The contents and rationale for the dist-info directory is described in PEP 376. It allows libraries such as distlib to find all installed distributions. I see no reason to discourage the creation of these directories in conda package and in fact would encourage a workflow that produced them. .egg file on the other hand can be harmful in conda packages and should be avoided.

[ocefpaf]

I've definitely forgotten to include pip in recipes before (when changing from setuptools to pip), and it fails quickly this way. If we switched to just pip, it'd just produce an empty package, which is definitely far less noticeable. A linter check would help with that, though, and we should probably add some kind of check for empty packages as well.

Second that! Also, many scientist use conda and pip as separate beasts (bad practice BTW) and they hardcode the patch of system's pip to their path. This would fail miserably there.

conda-build >=3.6 already has a check for empty packages https://github.com/conda/conda-build/blob/3.8.0/CHANGELOG.txt#L131-L133 smile

@mbargull empty package is only one of issues from that change. Even though I appreciate your PR this change is a hard no for me. It would break tons of stuff outside of conda-forge just to save a few keystrokes, that very few people actually type. Not worth it.

[jakirkham]

FWIW I can see value in cutting down the length of this command. It feels like a lot of cognitive load to be carrying around.

There is no cognitive load b/c people either copy-n-paste that. I would argue that should be in conda-skeleton to reduce it even further.

Copy and paste is the consequence of people acting under cognitive load. IOW people thinking, "What is this? Whatever I can't be bothered to think about it...". This will become more obvious later when we tell people they need to add even more options and they are confused. ( pypa/pip#5033 ) Ultimately this thinking is a serious concern for a community oriented package management project. So it really deserves consideration on our part in terms of what we are asking of people.

Given we already tell people to add pip as a build requirement explicitly, I think the paranoia of getting the wrong pip is unjustified.

Not a paranoia. A Fact. Again, conda-forge recipes are used outside of conda-forge and good practices do not hurt anyone.

Where pip will again be pulled in and put higher on the path. Sorry not seeing my original point refuted here.

[ocefpaf]

Sorry not seeing my original point refuted here.

Read it again with more care and thinking of others, maybe you will. Also, there are other options that we must pass that will always cause people to copy-n-paste. We cannot get rid of those too.

[mbargull]

@mbargull empty package is only one of issues from that change.

Sure, that comment was just to say "no need to do that part, already done".

Even though I appreciate your PR this change is a hard no for me. It would break tons of stuff outside of conda-forge just to save a few keystrokes, that very few people actually type. Not worth it.

I for one don't really care about two more "words"/few keystrokes. To me, it's just that I think pip is meant to be run that way. Hence I have a hard time to declare python -m pip part of "good practices".

If supporting setups outside of conda-forge is an objective for you, let me ask another way:
Calling python -m pip seems a bit arbitrary to me as a workaround -- is there anything else we can do to make this more robust? For conda-forge itself a linter check feels like the natural thing, but that, or course, does not cover external setups.

[ocefpaf]

To me, it's just that I think pip is meant to be run that way

One would argue that it is meant to be run that way in the context safely building packages vs a local user usage that you are describing.

[dopplershift]

python -m is the canonical way to ask python to resolve the path to a module and run it as a program, rather than relying on the shell $PATH (or equivalent) settings. That seems like exactly what you want in a case like this.

Speaking as a maintainer of several recipes, but not a core conda-forge member, I find the argument about "congnitive load" lack. I buy it for the other command link junk that may be needed, but python -m pip? No. It's the same as python -m pdb or python -m http.server.

[mbargull]

a local user usage that you are describing.

Never did, don't care about "local user" in this context.

python -m is the canonical way to ask python to resolve the path to a module and run it as a program, rather than relying on the shell $PATH (or equivalent) settings. That seems like exactly what you want in a case like this.

What we exactly want by invoking pip (irregardless of how), is just to run that helper application. While doing this -- and this is the important part why this discussion is controversial -- we want to be sure to operate on the Python environment that is intended to be used for the recipe.

---

So, I disagree with the explanations the both of you give. But this is nothing we have to dwell on.
You expect the "correct" python to be on PATH (which, btw, isn't just a "shell setting") and have that python configured to only pick up the associated pip. I expect the "correct" pip of the associated python to be on PATH.

---

TBH, as said before, if you think there is an uncertainty with this, then something like

  script: '"${PREFIX}/bin/pip" install --no-deps --ignore-installed .'  # [not win]
  script: '"%PREFIX%\Scripts\pip" install --no-deps --ignore-installed .'  # [win]

is the safer bet. If things like this are too error-prone or create too much code duplication, they can be put into helper scripts and such. (This is generally speaking -- (python -m) pip install --no-deps --ignore-installed . is really a borderline small example...)

[nicoddemus]

As a conda-forge contributor of several recipes, I wholeheartedly agree with @ocefpaf and @dopplershift: using python -m is the canonical way of asking Python to execute something and trivial to remember by any Pythonista, the other options are much harder to remember.

[ChrisBarker-NOAA]

python -m is the canonical way to ask python to resolve the path to a module and run it as a program, You expect the "correct" python to be on PATH (which, btw, isn't just a "shell setting") and have that python configured to only pick up the associated pip. That’s not an “expectation” your install is seriously broken if -m doesn’t work. Many people (including me) recommend the -m approach to newbies, too. There are a lot of errors when people “pip install” stuff, and then can’t import it. Particularly when someone has py2 and py3 I stalked at once. I expect the "correct" pip of the associated python to be on PATH. You can expect it all you want, and it may be a reasonable expectation within conda-build, but it’s not a safe expectation in general. In fact, I think the python.org installer (at least on OS-X) only installs “pip3”, and not “pip” with py3 by default. '"${PREFIX}/bin/pip" install --no-deps --ignore-installed .' # [not win] script: '"%PREFIX%\Scripts\pip" install --no-deps --ignore-installed .' # [win] is the safer bet. That is less safe than $PYTHON -m

…

-CHB

[mbargull]

and it may be a reasonable expectation within conda-build, but it’s not a safe expectation in general.

Yes, Conda/conda-build this is the context of this issue.

That is less safe than $PYTHON -m

This is untrue.

[mbargull]

Okay guys, if there is so much disapproval regarding that change, let's not repeat ourselves, leave this closed and get back to working on more pressing things.
Thank you all for your input!