.pxd files of dependency targeted recipe'd library not found

[ghost]

Description

It seems that when having two Cython libraries in the dependencies (both with a CythonRecipe with no special options) and one cimports things from the other (therefore requiring its .pxd files), something incorrect in python-for-android's Cython setup will break locating any of the .pxd files. I'm just getting errors like 'wobblui/cache.pxd' not found, 'wobblui/font/manager.pxd' not found, ... for all such "external" pxd files.

On desktop, the build works perfectly fine, even when installing the dependency target with the .pxd files remotely using pip3 install https://github.com/JonasT/wobblui/archive/master.zip - which should be how python-for-android fetches and installs it as well. I confirmed with updatedb && locate that the .pxd files are indeed installed, including in the python-for-android build environment. Just for some reason, Cython inside the python-for-android build doesn't "see" them.

TL;DR: .pxds of Cython dependency target libs aren't found. Breaks build, not good 😢

Versions

• Python: python3crystax, with python 3 build host
• OS: ubuntu in docker
• Kivy: not used
• Cython: 0.29

buildozer.spec

not used

[ghost]

Are recipe dependencies properly added to the site-packages paths before the respective next recipe is built/its setup.py is run? Are they available in the follow up recipe's setup.py virtualenv at all (including the Cythonizing virtualenv, if that's not the same one for p4a), not just at runtime later?

Because they would really need to be, otherwise it's no surprise this breaks. If this is the reason that'd be bad, because that makes any more performant cimports between Cython libraries per design impossible... (not only a perf issue, since my libraries are now also written to require this so I'd have to rip it out again even for the desktop version, or do branching code from hell to somehow work around this)

[ghost]

By the way as stated in chat, if this just involves fixing the path in p4a or something else along the lines, I'd be willing to make a pull request for this myself to help out. So if there are any suggestions on how to get this to work nicely, I'm all ears!

Edit: and one suggestion that was already made: modifying PATH/PYTHONPATH or something in the build virtualenv to include locations of all previous recipe install folders. although I wonder in case of no longer needing a recipe for zero-special-options cython projects (which is not the case now, but which I suggested as a future p4a change/fix) whether this would still work then due to the build order? does p4a build things in the required dependency order for non-recipe things?

[ghost]

Just to record the insights from chat, I checked the python-for-android code.
It has the following design errors:

• It just runs Cython manually, assuming all .pyx files in the project tree need to be built. (Not a bad guess, but simply not a necessarily correct assumption)
• It runs Cython with host python outside of any virtualenv. This is simply incorrect, because .pxd files from previous dependencies might be required for build. (This is why I got the error for which I made this ticket)

In overall the main problem seems to be that it manually runs Cython at all, when really the setup.py should get the correct env setup and be trusted to Cythonize. This ties in with the borked Cython defaults without a CythonRecipe to start with, because if all was set up correctly, none of this completely unnecessary manual Cythonize handling that makes all these wrong assumptions, or the need for a CythonRecipe when absolutely no special options are necessary, would be required in the first place.

I'll see if I can dig into the code and fix this

[ghost]

This issue will be resolved at least for projects which have a CythonRecipe purely to indicate Cython use (with no special options or patches for p4a needed) with this pull request: #1483 which makes CythonRecipes simply unnecessary for these projects, enabling proper regular setup.py processing which will make .pxd use work properly as well.

However, it will still be broken for projects using CythonRecipe. Not sure if it's worth fixing that, it'd need quite some changes on how that works. Better to motivate affected external projects to make using a recipe unnecessary in the first place, I would think. Probably not many who do cross-project .pxd includes anyway

[ghost]

Since using no CythonRecipe is an option now and that makes it work fine, this can probably be considered fixed sufficiently regarding the actual relevance in practice (as in it's still broken with CythonRecipe but with the other approach available it's probably not going to be an unsolvable issue for most people)