Linking against libraries from meson on Windows

[metab0t]

We need to modify the rpath of shared library on Linux and Mac, but there is no corresponding mechanism on Windows.

Are there any blockers preventing us from copying DLLs into wheels on Windows directly?

[rgommers]

Assuming you mean shared libraries that are built within the project that meson-python is invoked on, that may be feasible. The copying itself isn't the problem, DLLs can be put into .pkgname.mesonpy.libs just like on other platforms. The question to answer is how then to ensure that the Python extension modules that depend on those DLLs start working (xref gh-259). I'm not sure that there is a better answer than preloading them with, e.g., ctypes.WinDLL (as done in https://github.com/scipy/scipy/blob/main/tools/openblas_support.py#L217 for SciPy). And doing that by default is tricky - at least the question should be answered how that is done, and also what happens if then delvewheel is run on top of that.

[metab0t]

Thanks for detailed explanation.
The key is that how to make the packaged DLLs loadable from python (when they are not in PATH).
I think putting all generated DLLs of one package in the same directory will be feasible?
For example, we have the following directory structure (I build locally from https://github.com/mesonbuild/meson-python/tree/main/tests/packages/link-against-local-lib)

pypkg1
|-example.cp310-win_amd64.pyd
|-example.dll

Then import pypkg1.example as example runs without problem.

[rgommers]

I think putting all generated DLLs of one package in the same directory will be feasible?

It fixes the issue, but that's way too much of a restriction. Your average Python package will have multiple directories, and it can have extensions in each one of them. The RUNPATH solution for non-Windows platforms works no matter where you put your extension modules. For Windows we can achieve the same probably, but it needs some preloading solution and it's going to be uglier than using RUNPATH. Or maybe a shim dll per shared library that you want to ship:

pypkg1
|.pkgname.mesonpy.libs
  - example.dll
|pkgname
  |subdir 
    - example.cp310-win_amd64.pyd
    - example_mesonpy_dll_loader.dll   # generated dll, which has the job of preloading example.dll

[metab0t]

I understand your concerns. We need to make DLLs in .pkgname.mesonpy.libs loadable for all .pyd files in package files.
Maybe we can inject a one-line call to os.add_dll_directory to the top-level __init__.py of package?

[FFY00]

The issue is that AFAIK there's no way to do this without custom code on Windows.

To install the libraries inside the module, I think you can pass install: false to them, and then use them in https://mesonbuild.com/Python-module.html#install_sources.

For Windows we can achieve the same probably, but it needs some preloading solution and it's going to be uglier than using RUNPATH.

We could consider a solution similar than what we are doing for editable installs. But IMO this should be something provided by either Python, or the packaging ecosystem already.

[rgommers]

But IMO this should be something provided by either Python, or the packaging ecosystem already.

I would not expect anyone in Python packaging land to care, except scikit-build which will have the same type of user profile and demand for non-Python shared library support. So I think if we coordinate with them, that should suffice.

Maybe we can inject a one-line call to os.add_dll_directory to the top-level __init__.py of package?

I think that will work in principle, for Python >= 3.8 (which is enough for this feature). See the notes in https://github.com/MacPython/scipy-wheels/blob/054a522e893f489cd5ccb206f620c51395b09649/_distributor_init.py#L28-L59.

However, modifying a user's __init__.py is too magical. What if we just documented that users with shared libraries need to add this to their main __init__.py file, before importing any extension modules:

if os.name == 'nt':  # note: add clause for sys.version >= 3.8 if project still support py37
    mesonpy_dlldir = `../.pkgname.mesonpy.libs`  # fixme: use portable path specification here
    if os.exists(mesonpy_dlldir):  # directory may not exist, if we're building with `meson` directly
        os.add_dll_directory(mesonpy_dlldir)

That should be simple enough to not get wrong, and this only affects a small amount of users so I'd be happy enough with a doc-only solution here.

It only leaves out the case of no Python package but a standalone Python extension module which depends on a DLL, but in that case the user can do what @FFY00 suggests above and install both files to the same directory. That can be documented as well.

[metab0t]

The doc-only solution is OK, and meson-python should output warnings about this on Windows if DLLs are packaged.

[rgommers]

One more note on this: on Python 3.8 installed with Conda there may still be problems with os.add_dll_directory that have to be worked around with CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1 (see conda/conda#10897). This may be why SciPy is still preloading with ctypes.WinDLL.

[rgommers]

One alternative solution I just thought of is to ad a setting to allow running delvewheel on the wheel produced by meson-python, after meson-python is done with what it currently does, but before returning control to the build frontend (so inside the build_wheel hook). If it's opt-in and we make the package author responsible for adding delvewheel to their build dependencies on Windows (or we use a patchelf-like dependency insertion rule), that seems like a quite nice capability to have, and easier to implement than reinventing what delvewheel does or preloading with ctypes.

I believe this may work for most "build on Windows from source with a shared library" needs. The exception being if you are trying to ship a shared library in a wheel for consumption outside of that package (because the symbol mangling applied by delvewheel will make that impossible).

So a complete solution may be:

• include the shared library in the wheel just like on Linux/macOS
• have a setting for the package author to opt in to their preferred way of using that:
  • os.add_dll_directory (and then they have to add the .mesonpy.libs snippet themselves)
  • delvewheel usage