Cython .pxd imports + build isolation make dependency analysis super slow DESPITE hooks.get_requires_for_build_wheel()

[ghost]

Apologies, I have filed related tickets on this (see related tickets below), but there is still an angle to this that hasn't really been answered so hopefully you'll bear with me:

---

The problem:

Getting just the basic metadata, e.g. name & deps, of python packages can take a long time. hooks.get_requires_for_build_wheel() was an amazing addition to - in theory - speed up getting install_requires=() in some automated fashion (e.g. for use in a cross compilation environment where one might need to map out the dependencies to figure out if some of them need patches applied to work, as we do in https://github.com/kivy/python-for-android ) without waiting forever, by almost not processing anything of what would be normally required to fully build a wheel. E.g. it skips all the compilation...

... unless one uses .pxd cross-package imports, that is Cython cimport from an outside dependency. And here is why (I have explained this before, it's just another rehash):

The .pxd import targets have no nice way to of being specified, since setup_requires= breaks Cython so there's only pyproject.toml's build-system.requires which comes with a catch: now to do anything with this package, even just looking at its dependencies, all the .pxd import targets (which may be huge Cython libraries that take a long time to compile) need to be fully processed because neither pip nor pep517 can tell if such a buid-system.requires= dep is needed to fully build the wheel but not at all during hooks.get_requires_for_build_wheel().

Real world impact:

I have many packages/projects with large libs as .pxd import targets, and just trying to obtain the dependencies of these projects via hooks.get_requires_for_build_wheel()/pep517's envbuild takes multiple minutes. This is not a pretty situation and slows all my non-trivial project's builds for Android down considerably, since there python-for-android will scan all the dependencies. Since pip also doesn't currently reliably commit to a build order as seen in pypa/pip#6406 and also doesn't make them reliably available (just doesn't with build isolation) during build, these .pxd imports can't just be put into install_requires either which is one other alternative I investigated

It would therefore be nice if there was some mechanism to remove this .pxd import impact on build time.

I made these related tickets:

• over at Cython Would be nice if .pxd import could be done without requiring the depedency to be fully installed cython/cython#2924 (Cython ticket saying it would be nice to have another mechanism to make sure .pxd imports are around) but it seems like something that concerns packaging in general

• and here over at pip where we mostly derailed into the time impact caused by build isolation itself: Need for wheel caching: new pyproject.toml build isolation can cause excessive unnecessary compilation times with Cython & external .pxd import targets pip#6411 - which is very reasonable, because this does also affect any other sort of package that needs to be around during build AND at runtime as a dep, which would currently also be processed twice. (which would be fixed by wheel caching as proposed) It's just that it doesn't fix the issue that .pxd imports currently also slow down in case I just want to list the package dependencies and other metadata, which really isn't that nice

Ideas: (beyond already proposed wheel caching)

I'm really not sure what a good solution is, but the current situation just seems unsatisfactory. But what my mind comes back to is that build-system.requires basically mixes up "required to run setup.py" (-> needed for hooks.get_requires_for_build_wheel) and "only required to actually build the wheel" (-> not needed for hooks.get_requires_for_build_wheel). Only the latter applies a .pxd import target. So it seems to me that the semantics just don't match up.

What about another declarative pyproject.toml option just for "needed during build_ext, but not otherwise"? Or how about a pyproject.toml option for declaratively putting all install_requires deps into pyproject.toml so that the package doesn't even need to be processed? (Could be optional for all packages that don't need to runtime-adjust dependencies) There has to be something beyond just wheel caching that makes this situation make more sane

[pradyunsg]

There's some discussion at https://discuss.python.org/t/support-for-build-and-run-time-dependencies/1513.

[pradyunsg]

Closing since this discussion has moved to pypa/setuptools#1742.