Verify Wheel #694

dkeeney · 2019-09-27T03:11:42Z

The intent of the PR is to verify that the wheel package contains the right things.

…nto verifywheel

dkeeney · 2019-09-27T03:27:26Z

This is work in progress.
I found that the wheel did not contain the py unit tests. So I set about to copy the unit tests into directory where the wheel package is created (build/Release/distr/src/tests)

This involved:

moving setup.py from bindings/py/packaging to the top directory and eliminating the redirection call it was making. So the requirements.txt and the real setup.py are now both at the top of the repository.
moving the bindings unit tests (bindings/py/tests) to bindings/py/packaging/src/tests where it could be parallel with htm. Copying bindings/py/packaging/ to the distr folder gets both htm and tests folders with one copy.
copy py/tests to the distr folder (merging with the tests from packaging).
both copies had to filter out the pycache files.

That is as far as I have gotten so far. The next step is to build it and create the wheel like this for all platforms and see if everything is there in the wheel and that it can be installed using pip directly from the wheel.

Then I need to deal with the linux binary distribution thing.

breznak

This looks good, thanks for fixing it! Merge when CI passes.

So the requirements.txt and the real setup.py are now both at the top of the repository.

👍 I think it's good for the (py) packaging to have the standard layout.

Then I need to deal with the linux binary distribution thing.

yep, we'll have to think that out. Is it an issue only on Linux, so OSX,Win wheels would/do already work fine?

breznak · 2019-09-27T06:28:51Z

bindings/py/README.md

@@ -1,33 +1,23 @@

 ## Layout of directories:
 ```
-  Repository
-     bindings
+  REPO_DIR    external  -- cmake scripts to download/build dependancies


nit, looks like your editor adds weird line ends

@dkeeney please fix these

setup.py-old

breznak · 2019-09-27T06:38:43Z

Failing cmake call
https://github.com/htm-community/htm.core/pull/694/checks?check_run_id=238251520

dkeeney · 2019-09-27T14:11:15Z

No, I have lots of work to do on this before it is ready to merge.

dkeeney · 2019-09-27T16:49:03Z

Working on Windows....

I build with python setup.py install --user --force
I build the wheel with python setup.py bdist_wheel
Use pip to uninstall python -m pip uninstall htm.core
Use pip to install from wheel python -m pip install build/Release/distr/dist/htm.core-2.0.14-cp37-cp37m-win_amd64.whl

It says it installs Successfully installed htm.core-2.0.14
but I am getting this error:

ERROR: Error checking for conflicts.
Traceback (most recent call last):
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python37\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 3012, in _dep_map
    return self.__dep_map
  File "C:\Users\Dave\AppData\Local\Programs\Python\Python37\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2806, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

I have no idea what might be causing it.

dkeeney · 2019-09-27T19:14:30Z

Hmmm, I tried to install another unrelated package and got the same error. so maybe its something to do with the way I have python installed....and nothing to do with the wheel.

breznak · 2019-09-27T22:08:55Z

Use pip to install from wheel python -m pip install build/Release/distr/dist/htm.core-2.0.14-cp37-cp37m-win_amd64.whl

hey, we could probably test this as a part of the CI steps! The wheel is there, we can just uninstall the source-built library, and try if installing from whl works (+run a few tests)

for each platform, and run tests

…core into verifywheel

breznak · 2019-09-27T22:40:10Z

@dkeeney I added the CI check that the wheel is installable

dkeeney · 2019-09-27T23:20:29Z

After re-installing Python 3.7 and removing all other Pythons, I still get this error about conflicts.
But since it did say the install was successful. So I decided to try to install all dependencies as well.

pip install --user --force build/Release/distr/dist/htm.core-2.0.14-cp37-cp37m-win_amd64.whl

It gave me that same error as before. --ignoring that for now.
Then it gave me an error trying to install an egg . The error was cannot find the file specified. and the file was hexy-1.4.3-py3.7.egg
I can see that it downloaded and built the wheel for hexy but perhaps what it is actually doing is installing something else that needs hexy and hexy is not there yet.

Anyway, the point is....this still does not work yet.

dkeeney · 2019-09-27T23:23:22Z

Also, I cannot figure out how to run the .py unit tests without the full repository being present.
I can get them into the wheel but don't know how to run them.

dkeeney · 2019-09-27T23:58:04Z

@breznak you added python -m pip install --force build/Release/distr/dist/*.whl to the CI script. On Linux that works but on windows, the * is not expanded.

J:\Projects\AI\htmG>python -m pip install build/Release/distr/dist/*.whl
Requirement 'build/Release/distr/dist/*.whl' looks like a filename, but the file does not exist
*.whl is not a valid wheel filename.

I verified that the wheel does exists.
build/Release/distr/dist/htm.core-2.0.14-cp37-cp37m-win_amd64.whl

dkeeney · 2019-09-28T00:23:06Z

Traced down the Conflict error. It is a corruption in my site-packages directory caused by an earlier wheel install failure for which pip did not clean up correctly. Anyway, that one we can safely ignore.

dkeeney · 2019-09-28T00:29:44Z

In your CI script python -m pip install build/Release/distr/dist/*.whl

I added '--force' so that it will also verify that the wheel has the right requirements.txt and that they all can be installed.

That does seem to work here on windows if I expand the '*' before making the call.

dkeeney · 2019-09-28T00:32:17Z

python -m pytest bindings/py/packaging/tests

If I change this to python -m pytest bindings/py/tests
it does work :-)

breznak · 2019-09-28T08:48:56Z

Traced down the Conflict error. It is a corruption in my site-packages directory caused by an earlier wheel install failure for which pip did not clean up correctly. Anyway, that one we can safely ignore.

glad you had it figured. For these reasons, I always use python virtualenv (python -m venv), and if things break, I just rm the whole install.

breznak

Glad this works with the new changes!

breznak · 2019-09-28T08:51:03Z

.github/workflows/htmcore.yml

@@ -93,10 +93,12 @@ jobs:
        path: build/Release/distr/dist

    - name: Test python package is working
+      shell: bash


nice, this forces windows to act compatible with unixes, right? (is it using the WSL?)

.github/workflows/htmcore.yml

breznak · 2019-09-28T08:58:23Z

ls: build/Release/distr/dist/*.whl: No such file or directory

oh, ok, this is an unrelated err.

I've temporarily allowed some steps (that should be run only on tag/release) to run on each PR for debugging. But the creation of the whl is still only on tag.

for debugging this PR All FIXME should be addressed before merging

breznak

This is working now for all platforms 👍

Don't merge until the conditional filters for tags are re-enabled.

I'll try to add the manylinux building job in this PR

breznak · 2019-09-28T10:39:35Z

Run ls dist/
htm.core-2.0.16-cp37-cp37m-linux_x86_64.whl
htm.core-2.0.16-cp37-cp37m-macosx_10_14_x86_64.whl
htm.core-2.0.16-cp37-cp37m-win_amd64.whl
htm.core-2.0.16-py3.7-linux-x86_64.egg
htm.core-2.0.16-py3.7-macosx-10.14-x86_64.egg
htm.core-2.0.16-py3.7-win-amd64.egg
requirements.txt

why is egg still being created?

dkeeney · 2019-09-28T13:17:09Z

why is egg still being created?

It occurred to me in my sleep last night. The default for the setup.py setup() function is to generate an egg when we use the keyword install. So when we do python setup.py install --user --force it will create the egg. But if we do
python setup.py bdist_wheel --install --user --force instead to do the build it will generate a wheel as a product of the full build.

I wanted to try this out this morning.

dkeeney · 2019-09-28T13:20:55Z

python setup.py bdist_wheel --install --user --force

Oh, that should be python setup.py bdist_wheel --user --force

dkeeney · 2019-09-28T14:46:33Z

Oh, that should be python setup.py bdist_wheel --user --force

Well.... it does not like --user --force, so the overall build command is now python setup.py bdist_wheel
What this will NOT do is install locally. Apparently it is the local install that uses the .egg file.

The CI builds don't need the local egg install if we use the wheel to install.

dkeeney · 2019-09-28T14:51:13Z

My schedule:
I have some commitments this morning but I can work this after noon and Sunday. But Monday we are traveling to Algodones Mexico for dental work. May be gone 4-5 days.

breznak · 2019-09-28T21:41:22Z

It occurred to me in my sleep last night.

wow, nice #️⃣ do you dream of programming and electric sheep? :)

build command is now python setup.py bdist_wheel

I'll try to use this.

breznak · 2019-10-03T09:24:40Z

Uploading linux wheels to PYPI works!
https://github.com/htm-community/htm.core/pull/694/checks?check_run_id=246117912#step:8:15

breznak · 2019-10-03T11:33:28Z

Everything seems to work! Merging to test PYPI upload with a new tag

dkeeney added 2 commits September 26, 2019 20:09

Verify wheel

a49c851

Merge branch 'master' of https://github.com/htm-community/nupic.cpp i…

5a2c105

…nto verifywheel

dkeeney self-assigned this Sep 27, 2019

dkeeney added build in_progress labels Sep 27, 2019

fixed the path

df286b7

breznak previously approved these changes Sep 27, 2019

View reviewed changes

breznak added 3 commits September 28, 2019 00:28

CI: upload artifacts on each PR

160b52d

CI: test installing from wheel

ef23287

for each platform, and run tests

Merge branch 'verifywheel' of https://github.com/htm-community/nupic.…

83c6cd3

…core into verifywheel

breznak dismissed their stale review via 83c6cd3 September 27, 2019 22:37

dkeeney added 2 commits September 27, 2019 16:37

back out changes to add tests to wheel.

d6377b0

incorporating changes from beznak

a163d71

dkeeney added 4 commits September 27, 2019 19:24

adjusted path in test command

d3ae9c9

expanding the wildcard in github workflow

700f3b7

moving VERSION to .gitignore

00a9b5c

oops, restoring versioning.cmake

b814780

breznak previously approved these changes Sep 28, 2019

View reviewed changes

Temporarily disable/enable some filters/steps

fe0f733

for debugging this PR All FIXME should be addressed before merging

breznak dismissed their stale review via fe0f733 September 28, 2019 09:03

CI fix file path for wheel install

e65af32

breznak requested changes Sep 28, 2019

View reviewed changes

breznak added the python not py binding, but merge py code in repo label Sep 28, 2019

breznak added 3 commits October 1, 2019 20:40

try PYPI publish

3e9a1eb

CI: rename python wheel to manylinux1 for PYPI to accept it

f7aa7fa

CI: PYPI remove egg

105e297

CI: reenable IF conditions for tags only

f779ef8

breznak merged commit e6520f3 into master Oct 3, 2019

breznak deleted the verifywheel branch October 3, 2019 11:34

breznak mentioned this pull request Oct 3, 2019

CI: PYPI upload have only *.whl in dist/ #703

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify Wheel #694

Verify Wheel #694

dkeeney commented Sep 27, 2019 •

edited by breznak

Loading

dkeeney commented Sep 27, 2019

breznak left a comment

breznak Sep 27, 2019

breznak Sep 28, 2019

breznak commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

breznak commented Sep 27, 2019

breznak commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

breznak commented Sep 28, 2019

breznak left a comment

breznak Sep 28, 2019

breznak commented Sep 28, 2019

breznak left a comment

breznak commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

breznak commented Sep 28, 2019

breznak commented Oct 3, 2019

breznak commented Oct 3, 2019

Verify Wheel #694

Verify Wheel #694

Conversation

dkeeney commented Sep 27, 2019 • edited by breznak Loading

dkeeney commented Sep 27, 2019

breznak left a comment

Choose a reason for hiding this comment

breznak Sep 27, 2019

Choose a reason for hiding this comment

breznak Sep 28, 2019

Choose a reason for hiding this comment

breznak commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

breznak commented Sep 27, 2019

breznak commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 27, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

breznak commented Sep 28, 2019

breznak left a comment

Choose a reason for hiding this comment

breznak Sep 28, 2019

Choose a reason for hiding this comment

breznak commented Sep 28, 2019

breznak left a comment

Choose a reason for hiding this comment

breznak commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

dkeeney commented Sep 28, 2019

breznak commented Sep 28, 2019

breznak commented Oct 3, 2019

breznak commented Oct 3, 2019

dkeeney commented Sep 27, 2019 •

edited by breznak

Loading