Scipy, qutip, pandas problem

[martenrichter]

I have a bit weird dependency problem.
I have the following environment setup:

name: xeus-python-kernel
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - python=3.11
  - xeus-python
  - ipycanvas
  - ipywidgets
  - matplotlib
  - plotly
  - pandas
  - nbformat
  - numpy
  - anywidget
  - scipy
  - sympy
  - qutip

If I build it without qutip, everything works fine.
If I build it with qutip, loading

import plotly.express as px

fails, as it can not load pandas.

If I only do

import pandas

it fails with

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 1
----> 1 import pandas

File /lib/python3.11/site-packages/pandas/__init__.py:49
     46 # let init-time option registration happen
     47 import pandas.core.config_init  # pyright: ignore[reportUnusedImport] # noqa: F401
---> 49 from pandas.core.api import (
     50     # dtype
     51     ArrowDtype,
     52     Int8Dtype,
     53     Int16Dtype,
     54     Int32Dtype,
     55     Int64Dtype,
     56     UInt8Dtype,
     57     UInt16Dtype,
     58     UInt32Dtype,
     59     UInt64Dtype,
     60     Float32Dtype,
     61     Float64Dtype,
     62     CategoricalDtype,
     63     PeriodDtype,
     64     IntervalDtype,
     65     DatetimeTZDtype,
     66     StringDtype,
     67     BooleanDtype,
     68     # missing
     69     NA,
     70     isna,
     71     isnull,
     72     notna,
     73     notnull,
     74     # indexes
     75     Index,
     76     CategoricalIndex,
     77     RangeIndex,
     78     MultiIndex,
     79     IntervalIndex,
     80     TimedeltaIndex,
     81     DatetimeIndex,
     82     PeriodIndex,
     83     IndexSlice,
     84     # tseries
     85     NaT,
     86     Period,
     87     period_range,
     88     Timedelta,
     89     timedelta_range,
     90     Timestamp,
     91     date_range,
     92     bdate_range,
     93     Interval,
     94     interval_range,
     95     DateOffset,
     96     # conversion
     97     to_numeric,
     98     to_datetime,
     99     to_timedelta,
    100     # misc
    101     Flags,
    102     Grouper,
    103     factorize,
    104     unique,
    105     value_counts,
    106     NamedAgg,
    107     array,
    108     Categorical,
    109     set_eng_float_format,
    110     Series,
    111     DataFrame,
    112 )
    114 from pandas.core.dtypes.dtypes import SparseDtype
    116 from pandas.tseries.api import infer_freq

File /lib/python3.11/site-packages/pandas/core/api.py:1
----> 1 from pandas._libs import (
      2     NaT,
      3     Period,
      4     Timedelta,
      5     Timestamp,
      6 )
      7 from pandas._libs.missing import NA
      9 from pandas.core.dtypes.dtypes import (
     10     ArrowDtype,
     11     CategoricalDtype,
   (...)
     14     PeriodDtype,
     15 )

File /lib/python3.11/site-packages/pandas/_libs/__init__.py:18
     16 import pandas._libs.pandas_parser  # isort: skip # type: ignore[reportUnusedImport]
     17 import pandas._libs.pandas_datetime  # noqa: F401 # isort: skip # type: ignore[reportUnusedImport]
---> 18 from pandas._libs.interval import Interval
     19 from pandas._libs.tslibs import (
     20     NaT,
     21     NaTType,
   (...)
     26     iNaT,
     27 )

File interval.pyx:1, in init pandas._libs.interval()

File hashtable.pyx:1, in init pandas._libs.hashtable()

File missing.pyx:1, in init pandas._libs.missing()

File /lib/python3.11/site-packages/pandas/_libs/tslibs/__init__.py:40
      1 __all__ = [
      2     "dtypes",
      3     "localize_pydatetime",
   (...)
     36     "is_supported_dtype",
     37 ]
     39 from pandas._libs.tslibs import dtypes  # pylint: disable=import-self
---> 40 from pandas._libs.tslibs.conversion import localize_pydatetime
     41 from pandas._libs.tslibs.dtypes import (
     42     Resolution,
     43     periods_per_day,
     44     periods_per_second,
     45 )
     46 from pandas._libs.tslibs.nattype import (
     47     NaT,
     48     NaTType,
     49     iNaT,
     50     nat_strings,
     51 )

File conversion.pyx:1, in init pandas._libs.tslibs.conversion()

File qutip/core/data/base.pyx:6, in init qutip.core.data.base()

File /lib/python3.11/site-packages/qutip/__init__.py:33
     26     del matplotlib
     29 # -----------------------------------------------------------------------------
     30 # Load modules
     31 #
---> 33 from .core import *
     34 from .solver import *
     35 from .solver import nonmarkov

File /lib/python3.11/site-packages/qutip/core/__init__.py:2
      1 from .options import *
----> 2 from .coefficient import *
      3 from .qobj import *
      4 from .cy.qobjevo import *

File /lib/python3.11/site-packages/qutip/core/coefficient.py:3
      1 import numpy as np
      2 import scipy
----> 3 import scipy.interpolate
      4 import os
      5 import sys

File /lib/python3.11/site-packages/scipy/interpolate/__init__.py:167
      1 """
      2 ========================================
      3 Interpolation (:mod:`scipy.interpolate`)
   (...)
    165 (should not be used in new code).
    166 """
--> 167 from ._interpolate import *
    168 from ._fitpack_py import *
    170 # New interface to fitpack library:

File /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py:16
     14 from ._polyint import _Interpolator1D
     15 from . import _ppoly
---> 16 from .interpnd import _ndim_coords_from_arrays
     17 from ._bsplines import make_interp_spline, BSpline
     20 def lagrange(x, w):

File interpnd.pyx:1, in init scipy.interpolate.interpnd()

AttributeError: module 'scipy.spatial._qhull' has no attribute '__pyx_capi__'

so it seems that pandas invokes qutip (Why?), which invokes in turn scipy.interpolate. In this causes the other packages to fail.
Running

import scipy.interpolate

confirms it.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 import scipy.interpolate

File /lib/python3.11/site-packages/scipy/interpolate/__init__.py:167
      1 """
      2 ========================================
      3 Interpolation (:mod:`scipy.interpolate`)
   (...)
    165 (should not be used in new code).
    166 """
--> 167 from ._interpolate import *
    168 from ._fitpack_py import *
    170 # New interface to fitpack library:

File /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py:16
     14 from ._polyint import _Interpolator1D
     15 from . import _ppoly
---> 16 from .interpnd import _ndim_coords_from_arrays
     17 from ._bsplines import make_interp_spline, BSpline
     20 def lagrange(x, w):

File interpnd.pyx:1, in init scipy.interpolate.interpnd()

AttributeError: module 'scipy.spatial._qhull' has no attribute '__pyx_capi__'

Of course, scipy.interpolate is deprecated.
The questions is, why it is failing?
Are the deprecated functions missing in current scipy emscripten build and is this causing the problem? Or is a change upstream in qutip required?

[martenrichter]

Linking
jupyterlite/xeus-python-demo#46
which is probably related. But I think that this is a packaging issue.

[martinRenou]

@DerThorsten have you seen this?

AttributeError: module 'scipy.spatial._qhull' has no attribute '__pyx_capi__'

I have a feeling I've already seen this message before, I don't remember what we did about it or if we even fixed it.

[DerThorsten]

scipy.spatial._qhull was problematic before...as anything scipy related.
but I dont have a quick solution

[martenrichter]

With pyodide it works, I think they have a newer version?
If I understood it correctly emscripten-forge is using an pyodide build, correct?

[martenrichter]

Old fortran code seems to be one main problem. I just googled a bit, did you try https://github.com/r-wasm/flang-wasm?tab=readme-ov-file for building scipy?