Use pip to install packages in recipes generated by PyPI skeleton

[nicoddemus]

Using pip as recommended by conda-forge:

https://conda-forge.org/docs/meta.html#use-pip

Right now this is hard-coded to pip, but once flit gets some traction we might add support to it as well.

[mingwandroid]

FWIW I oppose this change. PIP doesn't care very much about external package building efforts.

[msarahan]

I don't think we have a ton of options here. Pip is the community standard. We'll either end up running pip or reimplementing its behavior. I'm not completely sure how https://www.python.org/dev/peps/pep-0517/ and https://www.python.org/dev/peps/pep-0518/ will change things. If they evolve to remove pip from the build process, great, but we really use pip or python setup.py install as installation tools, not as build tools, so those may not apply here at all.

PyPA has done some stuff lately that would make using pip for our purposes harder, but they have also been reasonable about discussing and fixing things when xoviat and others discussed it with them. I don't think the right course of action is to write them off - we have to get better at interop stuff, not stick our heads in the sand. We must talk more with PyPA, not less. We should discuss this with the team as a whole, but my vote right now is to go ahead with this change.

[mingwandroid]

OK fair enough, I just felt that these issues with build isolation should've been more obvious to them.

[msarahan]

Oh yes, I totally agree with that sentiment. That's why we need to be in better communication with them, to help them better understand the whole story.

[mingwandroid]

Anyway, we should hold off on this until a pip is released with the --build-isolation flag as we will need our recipes to include that.

[msarahan]

hmmm.... Yeah, that's interesting. We don't need that now, but we will when pip 10 is released. maybe a better route is to merge this now, but to chase the enabling of --build-isolation using pip.conf instead of a CLI flag, so the skeleton recipes won't need modification

[mingwandroid]

I'd rather go for the explicit is better than implicit (or hidden in another file) route personally.

[mingwandroid]

Also, adding pip always as a dependency makes the eco-system no longer bootstrappable (because you need pip to build some of pips dependencies).

[nicoddemus]

Anyway, we should hold off on this until a pip is released with the --build-isolation flag as we will need our recipes to include that.

I thought about that, but until that is released (which I concede should be soon) the recipe correctly reflects what should be done. We can always (and should) update this later when 10.0 is released.

Also, adding pip always as a dependency makes the eco-system no longer bootstrappable.

I think skeletons are meant as a starting point, not as a final recipe.

It seems the ecosystem is moving away from plain setuptools and more towards using external tool + metadata (pip and flit). The build/script generaeted by conda skeleton pypi in this PR merely acknowledges the current state of things, which is that pip is the recommended tool to install packages in a virtual environment and will work with most packages out there. Users need to adjust the options if a package has special requirements to build, but that goes without saying I believe.

[mingwandroid]

I think skeletons are meant as a starting point, not as a final recipe.

I disagree, for conda skeleton cran I have implemented an --update-policy argument with options of error, overwrite and merge and you can use this to update parts of a recipe. I believe that's a better approach. I've used it twice since then (running it on 350 packages at once) and it works well (I basically had no work to do, my build scripts, patches and extra dependencies were carried over and still applied). It's bespoke to R and a bit of a hack but should be made more general so all skeletons benefit from it.

Using the skeletons as a staring point aims too low.

[nicoddemus]

I disagree, for conda skeleton cran I have implemented an --update-policy argument with options of error, overwrite and merge and you can use this to update parts of a recipe.

I see, but IMHO for "conda skeleton pypi" it makes sense to use the recommended tool by PyPI to install packages.

[msarahan]

pip 10 got released. I'm afraid we now need to be much more careful about using pip. I do think that this is the right approach, but we must have the build isolation stuff figured out before we make this change.

[jjhelmus]

pip 10's build isolation is only triggered when a pyproject.toml file is present in the source. When build isolation is triggered it can be disabled by including the --no-build-isolation argument. There are some missing features in the current PEP 518 implementation in pip which are causing some projects to avoid or remove their pyproject.toml files.

Also, worthwhile to note that pip does not currently support the flit backend.

[nicoddemus]

Closing this as master already contains the current code. 🤗

Fixed in #2972

Thanks!

[github-actions]

Hi there, thank you for your contribution!

This pull request has been automatically locked because it has not had recent activity after being closed.

Please open a new issue or pull request if needed.

Thanks!