Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pyproject.toml with requires=["numpy"] #141

Closed
AxelHenningsson opened this issue Feb 17, 2022 · 15 comments · Fixed by #142
Closed

Add pyproject.toml with requires=["numpy"] #141

AxelHenningsson opened this issue Feb 17, 2022 · 15 comments · Fixed by #142
Assignees

Comments

@AxelHenningsson
Copy link
Contributor

As posted on stack there is perhaps a simple fix to make pip install work even if numpy is required in setup.py to compile?

Could perhaps be nice to try and add a pyproject.toml with requires = [ "numpy"] and see if the installation issue can be fixed. This is mostly annoying if anyone wants to depend on ImageD11, since they are then required to propagate the "please install numpy first documentation" into their own docs.

Best,
Axel

@jonwright
Copy link
Member

I think it needs all of setup_requires to go into the toml thing, so also six and setuptools? And perhaps a lot of other info too...

@AxelHenningsson
Copy link
Contributor Author

This simple pyproject.toml file seem to work like a charm 🎆 to pip install and run tests on my local windows 10 machine.

@AxelHenningsson
Copy link
Contributor Author

Seem Mac+Linux build is failing with this.

@AxelHenningsson
Copy link
Contributor Author

Found the logs: https://app.travis-ci.com/github/jonwright/ImageD11/jobs/560505555

seem to be failing on Xenial Linux with:
image
which seems like an old numpy version problem? Strange though, since the logs says that

pip3 install pytest numpy

was used to pre-install numpy before any pip installation.

@jonwright
Copy link
Member

I didn't get time to look in detail - no idea what goes wrong. Packaging has never worked for compiled code.

@AxelHenningsson
Copy link
Contributor Author

Closing this and keeping the rollback as the current solution for now.

@jadball
Copy link
Contributor

jadball commented Sep 20, 2022

Hi all, I'm running into problems with this now - I'm trying to set up poetry with my own pyproject.toml that includes the master branch of ImageD11 as a dependency:

# [...]
[tool.poetry.dependencies]
python = ">=3.8,<3.12"
ImageD11 = {git = "https://github.com/FABLE-3DXRD/ImageD11.git"}
# [...]

This installs fine with poetry (which uses pip under the hood) but I get problems when I try to import ImageD11 (see REPL below):

Python 3.8.14 (default, Sep 12 2022, 09:24:40) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 8.5.0
Python 3.8.14 (default, Sep 12 2022, 09:24:40) 
[GCC 12.2.0] on linux
import ImageD11
from ImageD11.grain import grain
Failed to import compiled C code for cImageD11 module
Are you running from a source directory?
Try something like:
   python -m pip install --editable .
or:
   python setup.py develop
or:
   python setup.py build_ext --inplace
Process finished with exit code 0

Any ideas? If I were using pip manually, I would solve this issue by installing ImageD11 from git with --no-use-pep517, but unfortunately these flags are not supported with poetry.

@jadball
Copy link
Contributor

jadball commented Sep 20, 2022

@jonwright @AxelHenningsson I believe I may have a solution here (albeit not a very good one) - the numpy error about mismatched versions is I think due to the following scenario:

  1. You have numpy somewhere on your PC (perhaps part of your root python install)
  2. This older version of numpy is used to compile the ImageD11._cImageD11 submodule
  3. Once the submodule is compiled, ImageD11 is installed - a different version of numpy can sometimes be installed here as a runtime requirement
  4. The version of numpy used to compile the C extensions is different to the one imported at runtime, hence the error

I made a fork here - if you force the same numpy version in build and runtime dependencies, it solves the problem for me.

The only issue then is that ImageD11 is tied to a specific numpy version, which causes issues with minimum python versions etc... a good excuse to drop Python 2 support? :)

@AxelHenningsson
Copy link
Contributor Author

Nice that someone is bravely still trying to solve this.

Did the build test on linux and max and the pytests pass with this?

Cheers
Axel

@jonwright
Copy link
Member

jonwright commented Sep 21, 2022

I thought the pep517 thing was about editable installs? Pinning the numpy version is not a good solution for the day you get another package that pins a different version.

What is the actual problem with your poetry thing? Does it give any logs that say what it did? Can you import ImageD11 and print ImageD11.__file__ to see where it is coming from? Perhaps it is related to setuptools versions?

From what I can see, it works on another computer here at ESRF (below). In the longer term, it might be useful to move the C code over to some other build system.

hpc5-0301:/data/id11/jon/scratch % python3 -m venv testenv
hpc5-0301:/data/id11/jon/scratch % source  testenv/bin/activate
(testenv) hpc5-0301:/data/id11/jon/scratch % which python
/gpfs/easy/data/id11/jon/scratch/testenv/bin/python
(testenv) hpc5-0301:/data/id11/jon/scratch % python -m pip install git+https://github.com/FABLE-3DXRD/ImageD11.git
Collecting git+https://github.com/FABLE-3DXRD/ImageD11.git
  Cloning https://github.com/FABLE-3DXRD/ImageD11.git to /tmp/pip-req-build-d6txr9j7
  Running command git clone -q https://github.com/FABLE-3DXRD/ImageD11.git /tmp/pip-req-build-d6txr9j7
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: setuptools in ./testenv/lib/python3.8/site-packages (from ImageD11==1.9.8) (44.0.0)
Collecting numpy
  Using cached numpy-1.23.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Collecting pytest
  Downloading pytest-7.1.3-py3-none-any.whl (298 kB)
     |████████████████████████████████| 298 kB 9.3 MB/s
Collecting fabio
  Downloading fabio-0.14.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB)
     |████████████████████████████████| 4.2 MB 54.4 MB/s
Collecting pyopengltk
  Downloading pyopengltk-0.0.4-py2.py3-none-any.whl (11 kB)
Collecting xfab>=0.0.4
  Downloading xfab-0.0.6-py2.py3-none-any.whl (68 kB)
     |████████████████████████████████| 68 kB 2.1 MB/s
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting matplotlib
  Downloading matplotlib-3.6.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.4 MB)
     |████████████████████████████████| 9.4 MB 21.3 MB/s
Collecting pyyaml
  Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
     |████████████████████████████████| 701 kB 36.6 MB/s
Collecting scipy
  Downloading scipy-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.4 MB)
     |████████████████████████████████| 43.4 MB 44.1 MB/s
Collecting h5py
  Using cached h5py-3.7.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
Collecting py>=1.8.2
  Downloading py-1.11.0-py2.py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 3.2 MB/s
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting pluggy<2.0,>=0.12
  Downloading pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting tomli>=1.0.0
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting attrs>=19.2.0
  Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 840 kB/s
Collecting packaging
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 1.8 MB/s
Collecting pyopengl
  Downloading PyOpenGL-3.1.6-py3-none-any.whl (2.4 MB)
     |████████████████████████████████| 2.4 MB 27.5 MB/s
Processing /home/esrf/wright/.cache/pip/wheels/d3/2a/90/c6700f618a0630c91b4c6d8ba6eb95fcc64369c5d94a27721b/PyCifRW-4.4.3-cp38-cp38-linux_x86_64.whl
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 26.5 MB/s
Collecting python-dateutil>=2.7
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.37.3-py3-none-any.whl (959 kB)
     |████████████████████████████████| 959 kB 35.1 MB/s
Collecting pyparsing>=2.2.1
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 3.4 MB/s
Collecting contourpy>=1.0.1
  Downloading contourpy-1.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (295 kB)
     |████████████████████████████████| 295 kB 55.1 MB/s
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting pillow>=6.2.0
  Downloading Pillow-9.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 41.6 MB/s
Building wheels for collected packages: ImageD11
  Building wheel for ImageD11 (PEP 517) ... done
  Created wheel for ImageD11: filename=ImageD11-1.9.8-cp38-cp38-linux_x86_64.whl size=470330 sha256=3a9b022bf456f584a5dacfc610fa59fadffbd0c1b3c7b3350325920bcabde705
  Stored in directory: /tmp/pip-ephem-wheel-cache-3k9wyebv/wheels/fd/d7/2d/0762bcce51fa2cad31a0215e6cd467a5bc51c920a5621dbbec
Successfully built ImageD11
Installing collected packages: numpy, py, iniconfig, pluggy, tomli, attrs, pyparsing, packaging, pytest, fabio, pyopengl, pyopengltk, six, pycifrw, xfab, kiwisolver, python-dateutil, fonttools, contourpy, cycler, pillow, matplotlib, pyyaml, scipy, h5py, ImageD11
Successfully installed ImageD11-1.9.8 attrs-22.1.0 contourpy-1.0.5 cycler-0.11.0 fabio-0.14.0 fonttools-4.37.3 h5py-3.7.0 iniconfig-1.1.1 kiwisolver-1.4.4 matplotlib-3.6.0 numpy-1.23.3 packaging-21.3 pillow-9.2.0 pluggy-1.0.0 py-1.11.0 pycifrw-4.4.3 pyopengl-3.1.6 pyopengltk-0.0.4 pyparsing-3.0.9 pytest-7.1.3 python-dateutil-2.8.2 pyyaml-6.0 scipy-1.9.1 six-1.16.0 tomli-2.0.1 xfab-0.0.6

(testenv) hpc5-0301:/data/id11/jon/scratch % python -c 'import ImageD11._cImageD11; print(ImageD11._cImageD11.__file__)'
/gpfs/easy/data/id11/jon/scratch/testenv/lib/python3.8/site-packages/ImageD11/_cImageD11.cpython-38-x86_64-linux-gnu.so

@jonwright jonwright reopened this Sep 21, 2022
@jadball
Copy link
Contributor

jadball commented Sep 21, 2022

I'm pretty embarrassed now - after working on this all day yesterday I now can't reliably recreate the error I was seeing with a minimal example.
The error I was getting was this:

------------------------------- Captured stdout --------------------------------
Failed to import compiled C code for cImageD11 module
Are you running from a source directory?
Try something like:
   python -m pip install --editable .
or:
   python setup.py develop
or:
   python setup.py build_ext --inplace
------------------------------- Captured stderr --------------------------------
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf

But now whatever I try, I can't get a simple Poetry project set up that fails reliably. It very well might have been due to some dodgy virtual environment stuff on my machine.

@jonwright
Copy link
Member

Problems recreating anything is the essence of python packaging 😄

I guess we could change that error message to print some more useful debugging information (cImageD11.file and sys,path perhaps).

The numpy versioning thing might go wrong if packages are added one by one, and a later package 'upgrades' or 'downgrades' something used by a previous one. It used to come up with conda when you start out with the fast intel/mkl version of numpy and then lose it when installing some package.

@jadball
Copy link
Contributor

jadball commented Sep 21, 2022

@jonwright Exactly - could we write a file during the build process that contains information about the version of numpy used to build ImageD11 (and its path, maybe)? This could help troubleshoot these issues. I'm pretty sure the issue arises when whatever packaging dependency solver you're using detects that there's a version of numpy already installs, builds ImageD11 with that, then updates it when the rest of the packages are installed.

@jonwright
Copy link
Member

Seems I suppressed the pyproject.toml in commit #96dada3, but I don't remember why. Putting it back and waiting for the CI...

@jonwright
Copy link
Member

It should be back #161, so I close the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants