h5py package

[talmo]

Hello! Thanks for this awesome project.

I'm eager to use h5py in a JupyterLite / xeus-python custom env, but h5py doesn't have a noarch package on conda-forge, nor a emscripten-wasm32 one here.

I don't see other issues tracking this particular widely used scientific package so I thought I'd start one to gather some info and see if anyone has any insights about bringing it into this ecosystem.

conda-forge

• Package available for linux-64, linux-aarch64, osx-64, osx-arm64, win-64, linux-ppc64le, win-32
• Feedstock repo

emscripten-forge / emscripten-wasm32

• hdf5 is already on emscripten-forge (recipe)
• PRs in this repo:
  • h5py wip #181 (by @DerThorsten)
  • adding h5py recipe #685 (by @shimwell)
  • Add h5py #1354 (by @talmo)

Other WASM

• Pkg h5py pyodide/pyodide#2411
• https://github.com/pyodide/pyodide/tree/3222f2ed2d57bd34a8b55276c0cb333997843c42/packages/h5py
• https://github.com/pyodide/pyodide/tree/3222f2ed2d57bd34a8b55276c0cb333997843c42/packages/libhdf5
• https://github.com/usnistgov/libhdf5-wasm
• https://github.com/usnistgov/h5wasm

Other discussions

• H5py not installable in jupyterlite xeus python, request for emscripten forge h5py/h5py#2338
• Feature request: out-of-tree Pyodide builds in CI for h5py (and libhdf5) h5py/h5py#2397
• adding openmc wasm executable shimwell/openmc-online-demo#1
• Patching scanpy for xeus scverse/scanpy#2667
• Troubleshooting an environment that won't build jupyterlite/xeus-python-demo#43

---

Hopefully this helps with findability for this particular package.

Anyone know where would be to best to check about current challenges remaining in #685 or if there may be other efforts to contribute to?

[shimwell]

I just updated the branch and retriggered the Ci for #685 but unfortunately the Ci fails with some erros that I don't know how to fix.

[talmo]

Hey @shimwell,

I was playing around with this and started a new draft PR that updates the recipe to the current rattler-build format here: #1354

It's actually passing basic checks and building locally, but now failing during imports 🤷

On the GH Actions runner, it's failing earlier in the build with:

 │ │   emcc -s MODULARIZE=1 -s LINKABLE=1 -s EXPORT_ALL=1 -s WASM=1 -s SIDE_MODULE=1 -sWASM_BIGINT -s SIDE_MODULE=1 -O2 -g0 -fPIC -I$PREFIX/include build/temp.u
 │ │ nknown-wasm32-cpython-311/home/runner/work/recipes/recipes/output/bld/rattler-build_h5py_1727601019/work/h5py/defs.o -L$PREFIX/lib -Wl,-R$PREFIX/lib -lhdf5
 │ │  -lhdf5_hl -o build/lib.unknown-wasm32-cpython-311/h5py/defs.cpython-311-wasm32-emscripten.so
 │ │   wasm-ld: error: unknown argument: -R$PREFIX/lib
 │ │   emcc: error: '$BUILD_PREFIX/bin/wasm-ld -o build/lib.unknown-wasm32-cpython-311/h5py/defs.cpython-311-wasm32-emscripten.so --whole-archive build/temp.unk
 │ │ nown-wasm32-cpython-311/home/runner/work/recipes/recipes/output/bld/rattler-build_h5py_1727601019/work/h5py/defs.o -L$PREFIX/lib -R$PREFIX/lib $PREFIX/lib/
 │ │ libhdf5.a $PREFIX/lib/libhdf5_hl.a -L$BUILD_PREFIX/opt/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic --no-whole-archive -mllvm -combine
 │ │ r-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --import-memory --strip-debug --export-dynamic --export-if-defined=main --
 │ │ export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-
 │ │ if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export-if-defined=__wasm_apply_data_relocs --export=__wasm
 │ │ _call_ctors --experimental-pic -shared' failed (returned 1)
 │ │   error: command '$BUILD_PREFIX/bin/emcc' failed with exit code 1

Update: This is now fixed using a monkey patch (from here) to pull out args like -R$PREFIX/lib and the thing compiles! Still fails when importing though (now matching my local behavior).

Logs:

 │ │ │ test_h5py.py 
 │ │ │ F
 │ │ │                                                            [100%]
 │ │ │ =================================== FAILURES ===================================
 │ │ │ _______________________________ test_import_h5py _______________________________
 │ │ │     def test_import_h5py():
 │ │ │ >       import h5py
 │ │ │ test_h5py.py:2: 
 │ │ │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 │ │ │ /lib/python3.11/site-packages/h5py/__init__.py:33: in <module>
 │ │ │     from . import version
 │ │ │ /lib/python3.11/site-packages/h5py/version.py:36: in <module>
 │ │ │     hdf5_version_tuple = _h5.get_libversion()
 │ │ │ h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
 │ │ │     ???
 │ │ │ h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
 │ │ │     ???
 │ │ │ h5py/h5.pyx:229: in h5py.h5.get_libversion
 │ │ │     ???
 │ │ │ h5py/defs.pyx:41: in h5py.defs.H5get_libversion
 │ │ │     ???
 │ │ │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 │ │ │ >   ???
 │ │ │ E   RuntimeError: Failed to walk error stack
 │ │ │ h5py/_errors.pyx:124: RuntimeError
 │ │ │ ----------------------------- Captured stderr call -----------------------------
 │ │ │ HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
 │ │ │   #000: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5.c line 1017 in H5open(): library initialization failed
 │ │ │     major: Function entry/exit
 │ │ │     minor: Unable to initialize object
 │ │ │   #001: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5.c line 250 in H5_init_library(): unable to initialize propert
 │ │ │ y list interface
 │ │ │     major: Function entry/exit
 │ │ │     minor: Unable to initialize object
 │ │ │   #002: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5Pint.c line 428 in H5P_init(): interface initialization failed
 │ │ │     major: Function entry/exit
 │ │ │     minor: Unable to initialize object
 │ │ │   #003: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5Pint.c line 497 in H5P__init_package(): can't register propert
 │ │ │ ies
 │ │ │     major: Property lists
 │ │ │     minor: Unable to register new atom
 │ │ │   #004: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5Pocpl.c line 163 in H5P__ocrt_reg_prop(): can't insert propert
 │ │ │ y into class
 │ │ │     major: Property lists
 │ │ │     minor: Unable to insert object
 │ │ │   #005: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5Pint.c line 2109 in H5P__register_real(): Can't insert propert
 │ │ │ y into class
 │ │ │     major: Property lists
 │ │ │     minor: Unable to insert object
 │ │ │   #006: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5Pint.c line 1222 in H5P__add_prop(): can't insert property int
 │ │ │ o skip list
 │ │ │     major: Property lists
 │ │ │     minor: Unable to insert object
 │ │ │   #007: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5SL.c line 1051 in H5SL_insert(): can't create new skip list no
 │ │ │ de
 │ │ │     major: Skip Lists
 │ │ │     minor: Unable to insert object
 │ │ │   #008: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5SL.c line 758 in H5SL__insert_common(): can't create new skip 
 │ │ │ list node
 │ │ │     major: Skip Lists
 │ │ │     minor: No space available for allocation
 │ │ │   #009: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5SL.c line 658 in H5SL__new_node(): memory allocation failed
 │ │ │     major: Skip Lists
 │ │ │     minor: No space available for allocation
 │ │ │   #010: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5FL.c line 2227 in H5FL_fac_malloc(): memory allocation failed
 │ │ │     major: Resource unavailable
 │ │ │     minor: No space available for allocation
 │ │ │   #011: /Users/thorstenbeier/micromamba/envs/emf/conda-bld/hdf5-0_1695653388114/work/src/H5FL.c line 253 in H5FL__malloc(): memory allocation failed for 
 │ │ │ chunk
 │ │ │     major: Resource unavailable
 │ │ │     minor: No space available for allocation
...