Testing application built with meson-python

[voidtrance]

I am working on an application that is Python-based (so it has a bunch of Python files). However, the application also implements a Python extension and some shared libraries.

The Python files import the Python extension as a module, which, in-turn, opens the shared libraries with dlopen.

I am not clear on how to test my application. When building the extension and libraries, they all get built into build/cp311 as described by the documentation. However, when I try to run my application from the Git repo, it fails to import the modules.

I also tried to run the application with PYTHONPATH=build/cp311/ but still it failed to load the modules.

How do I test my application?

Thank you.

[eli-schwartz]

In order for a python project to be importable when it consists of both .py files and compiled extensions, you have to do one of:

• have them be different top-level importable modules, no shared namespace
• install the project (at least in editable mode) and merge the source and build trees together

meson-python supports editable installation, so you can install it that way and then iterate on that for your tests. For a more in-depth approach, you can take a look at:

• scipy developer tooling: https://github.com/scipy/scipy/blob/main/dev.py
• standalone pluggable developer tooling framework, refactored based on the scipy one: https://pypi.org/project/spin/

[voidtrance]

In order for a python project to be importable when it consists of both .py files and compiled extensions, you have to do one of:

• have them be different top-level importable modules, no shared namespace

They are separate importable modules. However, the pure-Python module cannot find the C extension. This is the source tree after the mason-python build:

.
+- build/
|  +- cp311/
|     +- ...
|     +- src/
|        +- ext/
|           +- myext.cpython-311-x86_64-linux-gnu.so
+- src
|  +- ext
|     +- myext.c
+- module
|  +- __init__.py
+- main.py

main.py imports module needs to import myext. However, module can't find myext even with PYTHONPATH set.

• install the project (at least in editable mode) and merge the source and build trees together

I am not sure what this means. I am using editable builds but do you mean that after the build I would have to copy all of the pure-Python files into the build directory build/cp311?

[eli-schwartz]

I am not sure what this means. I am using editable builds but do you mean that after the build I would have to copy all of the pure-Python files into the build directory build/cp311?

Editable installs will create a hook loader that ensures that any time you import the project, regardless of current directory and regardless of $PYTHONPATH, you get the version you installed via the editable install. It also has some nice tricks such as automatically reloading any time you modify a source file -- because it will then run ninja to rebuild the C extensions before proceeding with the import, so you can edit source files and rerun your code / tests / interactive interpreter without manually rebuilding.

[eli-schwartz]

If PYTHONPATH contains both:

• $PWD (to execute the statement import module)
• $PWD/build/cp11/src/ext (to execute the statement import myext)

Then in principle I can't see what the problem would be when running directly from the source tree. Have you tried adding e.g. a debug print to main.py to have it show what it sees as sys.path when the failure occurs?