3d inference numba error

[romainGuiet]

Hi @Team,

I created a conda env using the empanada_gpu.yml (Win11)file :

name: "empanada_GPU"
channels:
  - conda-forge
  - defaults
  - ome
  - pytorch
dependencies:
  - python=3.9
  - cudatoolkit=11.3
  - pytorch:pytorch=1.12
  - qtpy
  - qtconsole
  - pyopengl
  - pip
  - ome::omero-py
  - pip:
      - "napari[all]"
      - ezomero
      - "empanada-dl"
      - "empanada-napari" 
      - "napari-omero"
prefix: "/opt/conda"

2D inference works well but I get an error when using 3D inference (with or without the zarr storage activated). Predictions step was done, and it seems to be at the end of the label preparation step
Please find here the error log empanada_error.txt

And here the exported conda env

name: D:\conda_envs\empanada_gpu
channels:
  - pytorch
  - ome
  - conda-forge
  - defaults
dependencies:
  - appdirs=1.4.4=pyh9f0ad1d_0
  - asttokens=2.0.5=pyhd8ed1ab_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - blas=2.115=mkl
  - blas-devel=3.9.0=15_win64_mkl
  - brotlipy=0.7.0=py39hb82d6ee_1004
  - bzip2=1.0.8=h8ffe710_4
  - ca-certificates=2022.6.15=h5b45459_0
  - certifi=2022.6.15=py39hcbf5309_0
  - cffi=1.15.1=py39h0878f49_0
  - charset-normalizer=2.1.0=pyhd8ed1ab_0
  - colorama=0.4.5=pyhd8ed1ab_0
  - cryptography=37.0.4=py39h7bc7c5c_0
  - cudatoolkit=11.3.1=h280eb24_10
  - debugpy=1.6.0=py39h415ef7b_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - executing=0.9.1=pyhd8ed1ab_0
  - expat=2.4.8=h39d44d4_0
  - freetype=2.10.4=h546665d_1
  - future=0.18.2=py39hcbf5309_5
  - gettext=0.19.8.1=ha2e2712_1008
  - glib=2.72.1=h7755175_0
  - glib-tools=2.72.1=h7755175_0
  - gst-plugins-base=1.20.3=he07aa86_0
  - gstreamer=1.20.3=hdff456e_0
  - icu=70.1=h0e60522_0
  - idna=3.3=pyhd8ed1ab_0
  - intel-openmp=2022.1.0=h57928b3_3787
  - ipykernel=6.15.1=pyh025b116_0
  - ipython=8.4.0=py39hcbf5309_0
  - ipython_genutils=0.2.0=py_1
  - jedi=0.18.1=py39hcbf5309_1
  - jpeg=9e=h8ffe710_2
  - jupyter_client=7.3.4=pyhd8ed1ab_0
  - jupyter_core=4.11.1=py39hcbf5309_0
  - krb5=1.19.3=h1176d77_0
  - lcms2=2.12=h2a16943_0
  - lerc=3.0=h0e60522_0
  - libblas=3.9.0=15_win64_mkl
  - libcblas=3.9.0=15_win64_mkl
  - libclang=14.0.6=default_h77d9078_0
  - libclang13=14.0.6=default_h77d9078_0
  - libdeflate=1.12=h8ffe710_0
  - libffi=3.4.2=h8ffe710_5
  - libglib=2.72.1=h3be07f2_0
  - libiconv=1.16=he774522_0
  - liblapack=3.9.0=15_win64_mkl
  - liblapacke=3.9.0=15_win64_mkl
  - libogg=1.3.4=h8ffe710_1
  - libpng=1.6.37=h1d00b33_3
  - libsodium=1.0.18=h8d14728_1
  - libtiff=4.4.0=h2ed3b44_1
  - libuv=1.44.2=h8ffe710_0
  - libvorbis=1.3.7=h0e60522_0
  - libwebp=1.2.3=h8ffe710_1
  - libwebp-base=1.2.3=h8ffe710_2
  - libxcb=1.13=hcd874cb_1004
  - libzlib=1.2.12=h8ffe710_2
  - lz4-c=1.9.3=h8ffe710_1
  - m2w64-gcc-libgfortran=5.3.0=6
  - m2w64-gcc-libs=5.3.0=7
  - m2w64-gcc-libs-core=5.3.0=7
  - m2w64-gmp=6.1.0=2
  - m2w64-libwinpthread-git=5.0.0.4634.697f757=2
  - matplotlib-inline=0.1.3=pyhd8ed1ab_0
  - mkl=2022.1.0=h6a75c08_874
  - mkl-devel=2022.1.0=h57928b3_875
  - mkl-include=2022.1.0=h6a75c08_874
  - msys2-conda-epoch=20160418=1
  - nest-asyncio=1.5.5=pyhd8ed1ab_0
  - omero-py=5.11.0=py39_0
  - openjpeg=2.4.0=hb211442_1
  - openssl=1.1.1q=h8ffe710_0
  - packaging=21.3=pyhd8ed1ab_0
  - parso=0.8.3=pyhd8ed1ab_0
  - pcre=8.45=h0e60522_0
  - pickleshare=0.7.5=py_1003
  - pillow=9.2.0=py39ha53f419_0
  - pip=22.2=pyhd8ed1ab_0
  - ply=3.11=py_1
  - prompt-toolkit=3.0.30=pyha770c72_0
  - psutil=5.9.1=py39hb82d6ee_0
  - pthread-stubs=0.4=hcd874cb_1001
  - pure_eval=0.2.2=pyhd8ed1ab_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pygments=2.12.0=pyhd8ed1ab_0
  - pyopengl=3.1.6=pyhd8ed1ab_1
  - pyopenssl=22.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pyqt=5.15.7=py39hb08f45d_0
  - pyqt5-sip=12.11.0=py39h415ef7b_0
  - pysocks=1.7.1=py39hcbf5309_5
  - python=3.9.13=h9a09f29_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=2_cp39
  - pytorch=1.12.0=py3.9_cuda11.3_cudnn8_0
  - pytorch-mutex=1.0=cuda
  - pywin32=303=py39hb82d6ee_0
  - pyyaml=6.0=py39hb82d6ee_4
  - pyzmq=23.2.0=py39he46f08e_0
  - qt-main=5.15.4=h467ea89_2
  - qtconsole=5.3.1=pyhd8ed1ab_0
  - qtconsole-base=5.3.1=pyha770c72_0
  - qtpy=2.1.0=pyhd8ed1ab_0
  - requests=2.28.1=pyhd8ed1ab_0
  - setuptools=63.2.0=py39hcbf5309_0
  - sip=6.6.2=py39h415ef7b_0
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.39.2=h8ffe710_0
  - stack_data=0.3.0=pyhd8ed1ab_0
  - tbb=2021.5.0=h2d74725_1
  - tk=8.6.12=h8ffe710_0
  - toml=0.10.2=pyhd8ed1ab_0
  - tornado=6.2=py39hb82d6ee_0
  - traitlets=5.3.0=pyhd8ed1ab_0
  - typing_extensions=4.3.0=pyha770c72_0
  - tzdata=2022a=h191b570_0
  - ucrt=10.0.20348.0=h57928b3_0
  - urllib3=1.26.11=pyhd8ed1ab_0
  - vc=14.2=hb210afc_6
  - vs2015_runtime=14.29.30037=h902a5da_6
  - wcwidth=0.2.5=pyh9f0ad1d_2
  - wheel=0.37.1=pyhd8ed1ab_0
  - win_inet_pton=1.1.0=py39hcbf5309_4
  - xorg-libxau=1.0.9=hcd874cb_0
  - xorg-libxdmcp=1.1.3=hcd874cb_0
  - xz=5.2.5=h62dcd97_1
  - yaml=0.2.5=h8ffe710_2
  - zeroc-ice36-python=3.6.5=py39h2e52968_7
  - zeromq=4.3.4=h0e60522_1
  - zlib=1.2.12=h8ffe710_2
  - zstd=1.5.2=h6255e5f_2
  - pip:
    - alabaster==0.7.12
    - albumentations==1.2.1
    - asciitree==0.3.3
    - attrs==21.4.0
    - babel==2.10.3
    - build==0.8.0
    - cachey==0.2.1
    - click==8.1.3
    - cloudpickle==2.1.0
    - commonmark==0.9.1
    - connected-components-3d==3.10.1
    - cycler==0.11.0
    - cztile==0.0.2
    - dask==2022.7.1
    - docstring-parser==0.14.1
    - docutils==0.19
    - empanada-dl==0.1.5
    - empanada-napari==0.2.1
    - ezomero==1.1.1
    - fasteners==0.17.3
    - fonttools==4.34.4
    - freetype-py==2.3.0
    - fsspec==2022.5.0
    - heapdict==1.0.1
    - hsluv==5.0.3
    - imagecodecs==2022.2.22
    - imageio==2.20.0
    - imagesize==1.4.1
    - importlib-metadata==4.12.0
    - jinja2==3.1.2
    - joblib==1.1.0
    - jsonschema==4.7.2
    - kiwisolver==1.4.4
    - llvmlite==0.39.0
    - locket==1.0.0
    - magicgui==0.5.1
    - markupsafe==2.1.1
    - matplotlib==3.5.2
    - napari==0.4.16
    - napari-console==0.0.4
    - napari-omero==0.2.0
    - napari-plugin-engine==0.2.0
    - napari-svg==0.1.6
    - networkx==2.8.5
    - npe2==0.5.2
    - numba==0.56.0
    - numcodecs==0.10.1
    - numpy==1.22.4
    - numpydoc==1.4.0
    - omero-rois==0.3.0
    - opencv-python==4.6.0.66
    - opencv-python-headless==4.6.0.66
    - pandas==1.4.3
    - partd==1.2.0
    - pep517==0.12.0
    - pint==0.19.2
    - pooch==1.6.0
    - psygnal==0.3.5
    - pydantic==1.9.1
    - pyrsistent==0.18.1
    - pytomlpp==1.0.11
    - pytz==2022.1
    - pywavelets==1.3.0
    - qudida==0.0.4
    - rich==12.5.1
    - scikit-image==0.19.3
    - scikit-learn==1.1.1
    - scipy==1.8.1
    - snowballstemmer==2.2.0
    - sphinx==5.1.1
    - sphinxcontrib-applehelp==1.0.2
    - sphinxcontrib-devhelp==1.0.2
    - sphinxcontrib-htmlhelp==2.0.0
    - sphinxcontrib-jsmath==1.0.1
    - sphinxcontrib-qthelp==1.0.3
    - sphinxcontrib-serializinghtml==1.1.5
    - superqt==0.3.3
    - threadpoolctl==3.1.0
    - tifffile==2022.5.4
    - tomli==2.0.1
    - toolz==0.12.0
    - torchvision==0.13.0
    - tqdm==4.64.0
    - typer==0.6.1
    - vispy==0.10.0
    - wrapt==1.14.1
    - zarr==2.12.0
    - zipp==3.8.1
prefix: D:\conda_envs\empanada_gpu

I compared my linux install which is working with:
- "empanada-dl"==0.1.4
- "empanada-napari"==0.2.0
- numba==0.55.2

but my windows install was crashing with :
- "empanada-dl"==0.1.5
- "empanada-napari"==0.2.1
- numba==0.56

so on windows , I tested a brand new conda env, pip install manually
- "empanada-dl"==0.1.4
- "empanada-napari"==0.2.0
- numba==0.55.2
and it's working (in CPU mode and GPU)!

I hope this describes enough the issue ...

Thank you for your help,

Cheers,

Romain

[conradry]

Hi Romain,

Thanks for reporting this. I noticed this bug myself yesterday. It's Windows specific, the newest release of empanada worked fine when I tested on Mac and Linux.

Numba is weird sometimes, it turns out this simple change fixed the bug in my testing. Update with pip install empanada-dl==0.1.6 and that ought to work.

[romainGuiet]

I confirm that the 0.1.6 solved the issue 👏 👏 👏
Thanks,
R