"numpy.ndarray size changed" error when installing pydivsufsort together with tensorflow==2.6.0

[Alexander-Serov]

There seems to be some incompatibility between the C-interface of pydivsufsort and that of numpy.

The observations are:

/usr/local/lib/python3.8/site-packages/mypack/test.py:5: in <module>
    from pydivsufsort import divsufsort, kasai
/usr/local/lib/python3.8/site-packages/pydivsufsort/__init__.py:2: in <module>
    from .stringalg import (
pydivsufsort/stringalg.pyx:1: in init pydivsufsort.stringalg
    ???
E   ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

After some extensive searching on the internet, I have found out it is likely related to the numpy's change of the C-interface:
https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
And in particular this answer, which also avoid updating numpy (it is imposed by my Tensorflow):
https://stackoverflow.com/a/69528768/2470337

So I was wondering if you have any idea of what this error may mean and how it could be corrected?

Ubuntu`pip freeze`

absl-py==0.15.0
astunparse==1.6.3
cachetools==4.2.4
certifi==2021.10.8
charset-normalizer==2.0.7
clang==5.0
eccodes==1.3.3
flatbuffers==1.12
gast==0.4.0
google-auth==2.3.2
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.41.1
h5py==3.1.0
idna==3.3
keras==2.6.0
Keras-Preprocessing==1.1.2
Markdown==3.3.4
numpy==1.19.5
oauthlib==3.1.1
opt-einsum==3.3.0
protobuf==3.19.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydivsufsort==0.0.5
requests==2.26.0
requests-oauthlib==1.3.0
rsa==4.7.2
six==1.15.0
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.6.0
tensorflow-estimator==2.6.0
termcolor==1.1.0
typing-extensions==3.7.4.3
urllib3==1.26.7
Werkzeug==2.0.2
wrapt==1.12.1

I only have this problem on Ubuntu systems. On Windows everything works just fine. My Windows pip freeze:

Windows `pip freeze`

absl-py==0.15.0
appdirs==1.4.4
argon2-cffi==21.1.0
astroid==2.8.4
astunparse==1.6.3
atomicwrites==1.4.0
attrs==21.2.0
autopep8==1.6.0
backcall==0.2.0
backports.entry-points-selectable==1.1.0
bleach==4.1.0
cachetools==4.2.4
certifi==2021.10.8
cffi==1.15.0
cfgrib==0.9.9.0
cfgv==3.3.1
cftime==1.5.1
charset-normalizer==2.0.7
clang==5.0
click==8.0.3
colorama==0.4.4
cycler==0.10.0
debugpy==1.5.1
decorator==5.1.0
defusedxml==0.7.1
dill==0.3.4
distlib==0.3.3
eccodes==1.3.3
emcee==3.1.1
entrypoints==0.3
et-xmlfile==1.1.0
execnet==1.9.0
filelock==3.3.1
findlibs==0.0.2
flatbuffers==1.12
func-timeout==4.3.5
gast==0.4.0
google-auth==2.3.1
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.41.1
h5py==3.1.0
identify==2.3.1
idna==3.3
importlib-metadata==4.8.1
iniconfig==1.1.1
ipykernel==6.4.2
ipython==7.28.0
ipython-genutils==0.2.0
ipywidgets==7.6.5
iso8601==0.1.16
isort==5.9.3
jedi==0.18.0
Jinja2==3.0.2
joblib==1.1.0
jsonschema==4.1.2
julian==0.14
jupyter-client==7.0.6
jupyter-core==4.8.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.2
kaleido==0.2.1
keras==2.6.0
Keras-Preprocessing==1.1.2
kiwisolver==1.3.2
lazy-object-proxy==1.6.0
lz4==3.1.3
Markdown==3.3.4
MarkupSafe==2.0.1
marshmallow==3.14.0
matplotlib==3.4.3
matplotlib-inline==0.1.3
mccabe==0.6.1
mistune==0.8.4
more-itertools==8.10.0
multiprocess==0.70.12.2
nbclient==0.5.4
nbconvert==6.2.0
nbformat==5.1.3
nest-asyncio==1.5.1
netCDF4==1.5.7
nodeenv==1.6.0
nose==1.3.7
notebook==6.4.5
numpy==1.19.5
oauthlib==3.1.1
openpyxl==3.0.9
opt-einsum==3.3.0
packaging==21.0
pandas==1.3.2
pandocfilters==1.5.0
parso==0.8.2
pathlib==1.0.1
pathos==0.2.8
patsy==0.5.2
pickleshare==0.7.5
Pillow==8.4.0
pipdeptree==2.2.0
platformdirs==2.4.0
plotly==5.3.1
pluggy==1.0.0
ply==3.11
pox==0.3.0
ppft==1.6.6.4
pre-commit==2.15.0
prometheus-client==0.11.0
prompt-toolkit==3.0.21
protobuf==3.19.0
py==1.10.0
pyarrow==4.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.8.0
pycparser==2.20
pydivsufsort==0.0.5
Pygments==2.10.0
pylint==2.11.1
Pympler==0.9
Pyomo==6.1.2
pyparsing==3.0.1
pyrsistent==0.18.0
pytest==6.2.5
pytest-forked==1.3.0
pytest-xdist==2.4.0
python-dateutil==2.8.2
pytz==2021.3
PyUtilib==6.0.0
pywin32==302
pywinpty==1.1.4
PyYAML==6.0
pyzmq==22.3.0
requests==2.26.0
requests-oauthlib==1.3.0
retrying==1.3.3
rsa==4.7.2
scikit-learn==1.0.1
scipy==1.7.1
seaborn==0.10.1
Send2Trash==1.8.0
singledispatchmethod==1.0
six==1.15.0
statsmodels==0.13.0
tenacity==8.0.1
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.6.0
tensorflow-estimator==2.6.0
termcolor==1.1.0
terminado==0.12.1
testpath==0.5.0
threadpoolctl==3.0.0
toml==0.10.2
tornado==6.1
tqdm==4.62.3
traitlets==5.1.1
typing-extensions==3.7.4.3
urllib3==1.26.7
virtualenv==20.9.0
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==2.0.2
widgetsnbextension==3.5.1
wrapt==1.12.1
xarray==0.19.0
xlrd==2.0.1
xmltodict==0.12.0
zipp==3.6.0

[louisabraham]

Hi, thank you for the report. Could you try installing with the --no-binary option? You'll need a compiler and to install the Cython dependency manually.

[Alexander-Serov]

Hi!

Thanks for the fast reply.

I have tried to, but the build fails with a compilation error I do not understand:

pip install pydivsufsort --no-binary pydivsufsort
Looking in indexes: https://pypi.org/simple
Collecting pydivsufsort
  Using cached pydivsufsort-0.0.5.tar.gz (222 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: wheel in /home/aserov/miniconda3/envs/sufsort-test/lib/python3.8/site-packages (from pydivsufsort) (0.37.0)
Requirement already satisfied: numpy in /home/aserov/miniconda3/envs/sufsort-test/lib/python3.8/site-packages (from pydivsufsort) (1.19.5)
Skipping wheel build for pydivsufsort, due to binaries being disabled for it.
Installing collected packages: pydivsufsort
    Running setup.py install for pydivsufsort ... error
    ERROR: Command errored out with exit status 1:
     command: /home/aserov/miniconda3/envs/sufsort-test/bin/python3.8 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r5eom001/pydivsufsort_190179392dd248ba9385fb7a80c60a4f/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r5eom001/pydivsufsort_190179392dd248ba9385fb7a80c60a4f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-fvuapwlq/install-record.txt --single-version-externally-managed --compile --install-headers /home/aserov/miniconda3/envs/sufsort-test/include/python3.8/pydivsufsort
         cwd: /tmp/pip-install-r5eom001/pydivsufsort_190179392dd248ba9385fb7a80c60a4f/
    Complete output (6 lines):
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --single-version-externally-managed not recognized
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/aserov/miniconda3/envs/sufsort-test/bin/python3.8 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r5eom001/pydivsufsort_190179392dd248ba9385fb7a80c60a4f/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r5eom001/pydivsufsort_190179392dd248ba9385fb7a80c60a4f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-fvuapwlq/install-record.txt --single-version-externally-managed --compile --install-headers /home/aserov/miniconda3/envs/sufsort-test/include/python3.8/pydivsufsort Check the logs for full command output.

[louisabraham]

I can reproduce on colab but don't understand either.

I was able to install on colab with

pip install git+https://github.com/louisabraham/pydivsufsort

I hope it can work for you!

[Alexander-Serov]

I have followed the advice on upgrading setuptools and wheels here, but nothing changed.

[louisabraham]

Also, can you provide a code example for your bug?

[Alexander-Serov]

Yes, thank you.

Sorry, yes, the code example is just importing:

from pydivsufsort import divsufsort, kasai

Before your last suggestion, on a pydivsufsort installed from a wheel, I was getting:

Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydivsufsort import divsufsort, kasai
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aserov/miniconda3/envs/sufsort-test/lib/python3.8/site-packages/pydivsufsort/__init__.py", line 2, in <module>
    from .stringalg import (
  File "pydivsufsort/stringalg.pyx", line 1, in init pydivsufsort.stringalg
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

However now, after your pip install git+https://github.com/louisabraham/pydivsufsort suggestion, the package has also installed, but the error is different:

Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydivsufsort import divsufsort, kasai
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aserov/miniconda3/envs/sufsort-test/lib/python3.8/site-packages/pydivsufsort/__init__.py", line 1, in <module>
    from .divsufsort import divsufsort, bw_transform, inverse_bw_transform, sa_search
  File "/home/aserov/miniconda3/envs/sufsort-test/lib/python3.8/site-packages/pydivsufsort/divsufsort.py", line 11, in <module>
    from .dll import libdivsufsort, libdivsufsort64
  File "/home/aserov/miniconda3/envs/sufsort-test/lib/python3.8/site-packages/pydivsufsort/dll.py", line 15, in <module>
    PATH_LIBDIVSUFSORT = next(PATH.glob("libdivsufsort.*"))
StopIteration

So I guess it is a good thing that it does not complain about the numpy interface anymore :)
It looks like it cannot find the original libdivsufsort files when installing this way.

Are you able to reproduce? I have the same result on 2 different Ubuntu machines.

[Alexander-Serov]

Btw, it seems like we have already discussed a similar topic here
#29
And the --single-version-externally-managed error was similar. :)

[louisabraham]

No, I cannot reproduce on Ubuntu: https://colab.research.google.com/drive/1jCyYm41NbYE5TjhV3-PKcJQToriXU0As?usp=sharing

Maybe you can try to clone and run those installation steps: https://github.com/louisabraham/pydivsufsort/blob/master/.travis.yml#L43-L47

They are used on linux arm64 because the standard pip way fails.

[seanlaw]

@Alexander-Serov Out of curiosity, what happens if you create a clean/separate environment and only install pydivsufsort without tensorflow (i.e., only install pydivsufsort and it's core dependencies and nothing else)? Does everything import properly and do the examples run?

[Alexander-Serov]

@seanlaw Yes, it does. Everything is fine. In my opinion, it is because numpy>=1.20 is installed then, which is probably the version for which pydivsufsort was compiled:

aserov@calysto:~/insight$ conda activate pydiv-only
(pydiv-only) aserov@calysto:~/insight$ pip install pydivsufsort
Looking in indexes: https://pypi.org/simple
Collecting pydivsufsort
  Using cached pydivsufsort-0.0.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)
Requirement already satisfied: wheel in /home/aserov/miniconda3/envs/pydiv-only/lib/python3.8/site-packages (from pydivsufsort) (0.37.0)
Collecting numpy
  Using cached numpy-1.21.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Installing collected packages: numpy, pydivsufsort
Successfully installed numpy-1.21.3 pydivsufsort-0.0.5

$ python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydivsufsort import divsufsort, kasai

However if I downgrade numpy to <1.20, then it fails (as suggested by the StackExchange discussion).

$ pip install "numpy<1.20"
Looking in indexes: https://pypi.org/simple
Collecting numpy<1.20
  Using cached numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl (14.9 MB)
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.3
    Uninstalling numpy-1.21.3:
      Successfully uninstalled numpy-1.21.3
Successfully installed numpy-1.19.5

$ python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydivsufsort import divsufsort, kasai
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aserov/miniconda3/envs/pydiv-only/lib/python3.8/site-packages/pydivsufsort/__init__.py", line 2, in <module>
    from .stringalg import (
  File "pydivsufsort/stringalg.pyx", line 1, in init pydivsufsort.stringalg
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Apparently, the wheel version available on pip has an implicit requirement of numpy>=1.20, which is not satisfied by the version required by tensorflow==2.6.0.

[Alexander-Serov]

But it's true, I don't know how it works on the colab. They have numpy==1.19.5, but there must be something special about their installation.

[Alexander-Serov]

I think it's that colab has python=3.7. Can you try on 3.8?

[louisabraham]

I know exactly why the program fails with the wheels: as you said pydivsufsort was compiled for a different version of NumPy.

Hence the best solution is to install directly from the sources and try to solve the path issue.

Did you try the other steps?

[Alexander-Serov]

I am looking into the .yml steps, yes.

[Alexander-Serov]

Well, I have played around a bit with the .yml and the path problem, and it seems like there are no

# unix
"libdivsufsort.*",
"libdivsufsort64.*",`

files in my installation folder.

Even if I git clone directly the repository for github and init the submodules, the libdivsufsort folder doesn't have files satisfying this pattern. Perhaps, I am missing smth?

$ ls
before-push.sh  dist      libdivsufsort  MANIFEST.in   pydivsufsort.egg-info  requirements.txt  tests
build.sh        examples  LICENSE        pydivsufsort  README.md              setup.py          tox.ini
$ ls libdivsufsort/
CHANGELOG.md  CMakeLists.txt  CMakeModules  examples  include  lib  LICENSE  pkgconfig  README.md  VERSION.cmake

So naturally, when you do a glob on the folder, there are no files matching the pattern. There is a folder libdivsufsort, but no files libdivsufsort.*.

[seanlaw]

@Alexander-Serov When installing from source, I have had some success doing the following:

# Starting from a clean environment
pip install cmake cython numpy
git clone --recurse-submodules https://github.com/louisabraham/pydivsufsort.git ./pydivsufsort.git
cd ./pydivsufsort.git
./build.sh
python -m pip install .

On my machine, this installed:

cmake=3.21.4
cython=0.29.24
numpy=1.21.3

Can you try this?

[Alexander-Serov]

Hi @seanlaw,

This does indeed work, thank you! The libdiv files are built and the correctly installed (with numpy==1.19.5).

For those trying after me, not that you have to cd out of the folder before trying, to use pydivsufsort. Otherwise, the import would fail.

So thanks again!

I just need to find a way now to integrate it into my pipelines!

[seanlaw]

This does indeed work, thank you! The libdiv files are built and the correctly installed

Great! I'm glad you figured it out.

For those trying after me, not that you have to cd out of the folder before trying, to use pydivsufsort. Otherwise, the import would fail.

Yes, this is because Python will try to import from the local pydivsufsort source directory rather than the version that you just installed inside of site-packages.

[Alexander-Serov]

Thanks for the fast replies. The issue is solved, I am closing!

[louisabraham]

Thank you @seanlaw !!!

[Alexander-Serov]

By the way, is there a way I can contribute to the repository, so that it also builds automatically a version for this numpy version? That could help other people facing the problem and me with integration into my pipelines.

[louisabraham]

No I don't think there is a possibility for specifying requirements like this on pypi and have pre built versions. So the source install should be used, by using the no binary option, and will work once Sean is done with his changes.

…

On Fri, Nov 5, 2021, 11:16 Alexander Serov ***@***.***> wrote: By the way, is there a way I can contribute to the repository, so that it also builds automatically a version for this numpy version? That could help other people facing the problem and me with integration into my pipelines. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#30 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEQQFIK64LC7W26Q742JG3UKOVG5ANCNFSM5G7KCJEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[seanlaw]

I know exactly why the program fails with the wheels: as you said pydivsufsort was compiled for a different version of NumPy.

@louisabraham I am not well educated on this so please educate me. Is there a reason why this package needs to be tied to a specific version of NumPy? I understand having a minimum supported version but I didn't realize the wheels were pinned to a specific NumPy version? To ask the question differently, can you think of a way to resolve this so that the wheels are compatible with versions of NumPy that are newer than the one packaged with the wheels? But, firstly, I'd like to educate myself on why this is a problem as I'm not well versed here.

[Alexander-Serov]

@louisabraham Isn't it possible to have one pre-compiled version of pydivsufsort for strictly numpy>=1.20 and one for strictly numpy<1.20. This could allow us to specify which wheel should be used, no?

@seanlaw The problem seems to come from a change in the numpy C interface in the version 1.20. My understanding is that apparently some part of a compiled binary code (perhaps from divsufsort) is included in the wheel, so when you launch it with a different numpy C interface, this binary part is no longer compatible.

[Alexander-Serov]

Where can I follow @seanlaw 's changes?

[seanlaw]

The problem seems to come from a change in the numpy C interface in the version 1.20. My understanding is that apparently some part of a compiled binary code (perhaps from divsufsort) is included in the wheel, so when you launch it with a different numpy C interface, this binary part is no longer compatible.

I have a naive question. Instead of focusing on the specific NumPy version, is there something that change or add like a try/except or if/else that can accommodate any version of NumPy (above a minimum version)? @Alexander-Serov perhaps this is something that you can help dig into? That is, how can we generalize the code so that it is compatible with any version of NumPy but without having to explicitly specify or hardcode the NumPy version.

Where can I follow @seanlaw 's changes?

What I am doing isn't necessarily focused on the NumPy version but to figure out how to separate the frontend from the backend. Some work is starting in #34

[Alexander-Serov]

I don't think we are talking about the same things here.
Actually specifying the numpy version on build is already a way to do the try/except you are talking about. In fact, it is pip that does the try/except and finds the last version of the package that satisfies other requirements (e.g. numpy). We release packages internally at our company and this works pretty fine. [This is also exactly the way pip chooses the right numpy and pandas version when you install tensorflow. If tensorflow does it, perhaps pydivsufosrt coudl as well?]

So basically you would need to just release a, say, pydivsufsort==0.5.1 with numpy<1.20 in its requirements and pydivsufsort==0.6.0 with numpy>=1.20 in the requirements and the builds and distribution of the right versions will be automatic thanks to pip.

[louisabraham]

I don't think a try except would do anything.

@Alexander-Serov it's just not how pip works. pip just tries to download the latest version and apply its requirements.

The base of the issue is this: https://numpy.org/devdocs/release/1.20.0-notes.html#size-of-np-ndarray-and-np-void-changed
combined with Cython compilation.

However, this blog post is interesting: https://ymd_h.gitlab.io/ymd_blog/posts/numpy_1_20_abi_change/

It indicates that we might be able to deliver pre-built packages that are compatible with all versions of NumPy if we build them with the older API.

I also found this: https://stackoverflow.com/a/40846742/5133167

Apparently, Cython does its job too well and the exception is just a warning that we can ignore in the specific case of numpy that ensures binary compability.

I think the second solution is a better idea. Do you want to propose a PR?

[Alexander-Serov]

@louisabraham I am positive that pip does try to find the right version now. This is an example of what I get when pip finds incompatible requirements in the last version:

Collecting pytest-cov
  Downloading pytest_cov-2.12.0-py2.py3-none-any.whl (20 kB)
  Downloading pytest_cov-2.11.1-py2.py3-none-any.whl (20 kB)
  Downloading pytest_cov-2.11.0-py2.py3-none-any.whl (20 kB)
  Downloading pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)
  Downloading pytest_cov-2.10.0-py2.py3-none-any.whl (19 kB)
  Downloading pytest_cov-2.9.0-py2.py3-none-any.whl (19 kB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pytest-cov to determine which version is compatible with other requirements. This could take a while.
  Downloading pytest_cov-2.8.1-py2.py3-none-any.whl (18 kB)
  Downloading pytest_cov-2.8.0-py2.py3-none-any.whl (18 kB)
  Downloading pytest_cov-2.7.1-py2.py3-none-any.whl (17 kB)
  Downloading pytest_cov-2.7.0-py2.py3-none-any.whl (17 kB)
  Downloading pytest_cov-2.6.1-py2.py3-none-any.whl (16 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking

I think it worked like you say a year ago but has since evolved.

But I don't insist.

[louisabraham]

Oh very interesting!!! Sorry, I didn't know, thanks for the information.

However I'm not sure deploying two versions each time with two different sets of requirements is the best idea.

It seems that the exception can actually be ignored, so this is definitely the best way.

[Alexander-Serov]

As for the second solution you are suggesting, it looks interesting. But in the stackexchange post you are referring, the message is indeed a warning and the code seems to run. While in my case, it is clearly a ValueError and a specific size mismatch is cited (80 v. 88). Do you think it still applies?

Also, following on the same SE solution you cited, one of the conclusions of the scipy authors seems to be:

[Rebuilding everything against current numpy is] not a feasible solution, and certainly shouldn't be necessary. Scipy (as many other packages) is compatible with a number of versions of numpy. So when we distribute scipy binaries, we build them against the lowest supported numpy version (1.5.1 as of now) and they work with 1.6.x, 1.7.x and numpy master as well.

If we applied it here, again, we would return to the idea of building against the lowest valid supported version of numpy. I would tend to interpret this as numpy<1.20, but I am guessing you would disagree? :)

[Alexander-Serov]

Oh very interesting!!! Sorry, I didn't know, thanks for the information.

However I'm not sure deploying two versions each time with two different sets of requirements is the best idea.

It seems that the exception can actually be ignored, so this is definitely the best way.

What I meant is you don't need to deploy 2 versions every time. Just deploy once a version 0.5.1 with a strict requirement of numpy<1.20 and build against it. And then, starting from pydivsufsort>=0.6.0 specify and always build a single version with numpy>=1.20. The pydivsufsort~=0.5 was perfectly fine for the last 5 years since its release, so it could be just kept compatible with numpy<1.20. And all the new developments would support >=1.20.

[louisabraham]

We have a possible solution that would allow everyone to benefit from the last updates so let's try it!

I don't want to support 2 versions of the package if I can avoid it.

[louisabraham]

The SO question definitely has an error. We can try it, else I don't mind using an older version of numpy, I'm using basic features.

[Alexander-Serov]

Which possible solution of the 2 I mentioned do you have in mind? :)

Otherwise, I can make a PR, no problem.

I will also reopen this issue, just to keep track.

[Alexander-Serov]

On StackExchange, they are able to ignore the warning exactly because it is a warning, I feel (https://stackoverflow.com/a/40846742/2470337):

import warnings
warnings.filterwarnings("ignore", message="numpy.dtype size changed")
warnings.filterwarnings("ignore", message="numpy.ufunc size changed")

Their original message:

/usr/local/lib/python2.7/dist-packages/sklearn/svm/base.py in <module>()
      6 from abc import ABCMeta, abstractmethod
      7
----> 8 from . import libsvm, liblinear
      9 from . import libsvm_sparse
     10 from ..base import BaseEstimator, ClassifierMixin

__init__.pxd in init sklearn.svm.libsvm (sklearn/svm/libsvm.c:10207)()

RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 80

says warning.

[louisabraham]

it's actually a cython problem so I asked there cython/cython#4452

[Alexander-Serov]

Good idea! Let's see what they say.

[seanlaw]

This is a great discussion and I am learning a lot so thank you both!

However I'm not sure deploying two versions each time with two different sets of requirements is the best idea.

I agree that we'd want to avoid deploying two versions. I hope the Cython folks have a solution that solves all of our needs! 🤞

[seanlaw]

What I am doing isn't necessarily focused on the NumPy version but to figure out how to separate the frontend from the backend. Some work is starting in #34

@Alexander-Serov Is this something that you have experience with or would be willing to help assist with? It'll take some time for me to wrap my head around it and then find an opportunity to address it. It's at least somewhat stable but far from ready.

[da-woods]

Just to copy what I posted on the Cython issue:

It's an error if the size goes down and a warning if the size goes up. I believe the right thing to do is to build it with the earliest version of Numpy that you aim to support (so if you build it with 1.19 it should work with both, and if you can go back further that's even better).

[Alexander-Serov]

Perfect! That's a simple solution.

From what I've seen, pydiv does not use any new new numpy functionality. How low can the numpy version we can build it with then be, you think?

[Alexander-Serov]

@seanlaw Yes, I've seen issue #34 but unfortunately I don't have much experience with frontend splitting or multi platform building. Normally other people do that at my company. I will have a look whether someone has done something similar but cannot guarantee. 😔

[Alexander-Serov]

Apparently the proposed numpy version is not optimal for some configs. Re-opening until built correctly.

[Alexander-Serov]

Oops Cannot re-open. :)

[seanlaw]

Apparently the proposed numpy version is not optimal for some configs. Re-opening until built correctly.

Please see #35 (comment)