Self-contained OCP wheel to enable installation via pip #1048
Comments
|
Thanks a lot for this work. I created a pyproject.toml that uses this ocpvtk wheel. I tested it on Linux but it should work on other platforms. This way, CadQuery can be easily installed and published with Poetry: For now it uses url of a local server for wheels, but once the weels will be generated by the CI, a CadQuery package could be published (by the CI) on Pypi (ie. on https://pypi.org/project/cadquery/), and I could create a PR to add this pyproject.toml. |
|
I published the docker image on the Docker hub: So the Linux wheel can be retrieved with: I updated the Dockerfile in order to download Miniforge during the build. At first, my motivation for this was to enable automated builds on the Docker Hub in order to automatically build the image on each release, but sadly this option is now only available to paid plans... But at least, if anyone want to try the wheel, it's easier to retrieve it now. :) |
|
I'm trying to configure the CI. For now I was able to build the Linux wheel, it's a work in progress. |
|
I've done some work on creating a CQ wheel with conda-press. I've got other things to do first, but I'll see if I can combine this work with mine to get something working in CI.
As noted, we'll definitely need to address this in CI before we can push to PyPi. |
|
Tadaa:
ocp wheels for Windows, macOS and Linux, and python 3.8, 3.9 and 3.10, generated by the CI! All of them has been successfully tested in the CI with I just created a request to Pypi in order to ask for an increase file size limit (default is 100MB), I'm waiting for their response. 🙂 |
|
The ocp-vtk wheels generated by the CI are now available in a github release, which means that they can be downloaded programmatically. There is some work to do to improve the installation process, but you can still give a try: Where Each wheel is tested on the CI, but we will be happy to know if it works on your platform, in particular if you use Windows or macOS. |
|
Just create a venv first, to avoid messing up you system: *Temporary name, I would suggest to move on |
|
I'm used the ocp-vtk wheel to set up a CadQuery development environment on Linux and it works well. I've also tested the cadquery2 wheel in a couple of different ways, but only on Linux. I'll post this issue to the Discord server to see if we can get some testers on Windows and MacOS. |
|
I have windows and will test this tonight and report back. |
|
Ok, I have tested and can't get it to work. Summary: All failed with this error: More Detail is below . Using conda, installation succeeded, but with pip warnings. Ultimately the installation failed: it could not find OCP dlls: ` To activate this environment, use To deactivate an active environment, use (base) C:\Users\davec\gitwork\CQ-editor>conda activate cq2 (cq2) C:\Users\davec\gitwork\CQ-editor>pip install cadquery2 (cq2) C:\Users\davec\gitwork\CQ-editor>python -c "import cadquery2 as cq; print(cq.Workplane('XY').box(1,2,3).toSvg())" (cq2) C:\Users\davec\gitwork\CQ-editor>python
I upgraded pip, and forced re-installation, and got the same error: `(cq2) C:\Users\davec\gitwork\CQ-editor>pip install --force-reinstall cadquery2 (cq2) C:\Users\davec\gitwork\CQ-editor>import cadquery2 as cq (cq2) C:\Users\davec\gitwork\CQ-editor>python
I retried using a virtual env, and got the same result: C:\Users\davec\gitwork\CQ-editor>python -m venv c:/tmp/cq2 C:\Users\davec\gitwork\CQ-editor>cd c:/tmp c:\tmp>ls c:\tmp>dir Directory of c:\tmp 04/27/2022 05:58 PM .04/03/2022 02:16 PM 23,333 basicsplitflap-brd.svg 04/03/2022 01:32 PM 22,331 basicsplitflap-B_Cu.svg 04/03/2022 01:32 PM 2,019 basicsplitflap-Edge_Cuts.svg 12/06/2020 07:48 PM battlepoint 04/27/2022 05:58 PM cq2 04/03/2022 01:36 PM 27,965 drawing.hpgl 04/03/2022 01:36 PM 6,649 drawing.pdf 04/03/2022 01:35 PM 27,949 drawing.svg 10/16/2021 03:48 PM 155,679 Face.SLDPRT 12/23/2021 10:12 AM flash_sound 04/03/2022 01:52 PM 1,542,455 g1188.png 04/03/2022 02:10 PM 58,539 g535.png 04/03/2022 02:13 PM 59,320 g536.png 03/14/2021 07:42 PM 63 intro.py 03/14/2021 07:50 PM pgz 06/12/2021 04:41 PM 1,502 SeeduinoBreakout-Edge_Cuts.svg 03/14/2021 07:28 PM 762 test.py 04/03/2022 02:22 PM 615,400 test_board.lbrn 12/23/2021 10:35 AM 340 tf_sound.log 15 File(s) 2,544,306 bytes 5 Dir(s) 39,225,929,728 bytes free c:\tmp>cd cq2 c:\tmp\cq2>dir Directory of c:\tmp\cq2 04/27/2022 05:58 PM .04/27/2022 05:58 PM .. 04/27/2022 05:58 PM Include 04/27/2022 05:58 PM Lib 04/27/2022 05:58 PM 89 pyvenv.cfg 04/27/2022 05:58 PM Scripts 1 File(s) 89 bytes 5 Dir(s) 39,225,929,728 bytes free c:\tmp\cq2>Scripts/activate c:\tmp\cq2>cd Scripts c:\tmp\cq2\Scripts>activate (cq2) c:\tmp\cq2\Scripts>pip install --upgrade pip (cq2) c:\tmp\cq2\Scripts>pip install cadquery2 (cq2) c:\tmp\cq2\Scripts>python
the dlls are definitely installed, I can see them in what I think is the right spot: Directory of c:\tmp\cq2\Lib\site-packages 04/27/2022 06:00 PM .04/27/2022 05:58 PM .. 04/27/2022 05:59 PM 4,603 .load-order-ocp_vtk-7.5.3.0 04/27/2022 06:00 PM cadquery2 04/27/2022 06:00 PM cadquery2-2.1.1.dist-info 04/27/2022 05:59 PM 10,752 charset-f1ff420e.dll 04/27/2022 05:58 PM 126 easy_install.py 04/27/2022 05:59 PM ezdxf 04/27/2022 06:00 PM ezdxf-0.17.2.dist-info 04/27/2022 05:59 PM 634,880 freetype-4118af61.dll 04/27/2022 05:59 PM 95,744 gl2ps-d5be8415.dll 04/27/2022 05:59 PM 451,584 glew32-45501540.dll 04/27/2022 05:59 PM 739,840 hdf-839eff43.dll 04/27/2022 05:59 PM 3,164,672 hdf5-4fe83a80.dll 04/27/2022 05:59 PM 117,248 hdf5_hl-1e917a47.dll 04/27/2022 05:59 PM 939,008 iconv-cfade252.dll 04/27/2022 05:59 PM 57,856 jbig-283e2ee0.dll 04/27/2022 05:59 PM 227,840 jsoncpp-74e0510f.dll 04/27/2022 05:59 PM 385,024 Lerc-d98a2f21.dll 04/27/2022 05:59 PM 187,904 libbz2-bf72f086.dll 04/27/2022 05:59 PM 5,070,848 libcrypto-3-x64-79e66d5d.dll 04/27/2022 05:59 PM 513,024 libcurl-4715d16f.dll 04/27/2022 05:59 PM 66,048 libdeflate-b8e589f8.dll 04/27/2022 05:59 PM 407,040 libexpat-f770ecdf.dll 04/27/2022 05:59 PM 113,664 liblz4-b9e0f074.dll 04/27/2022 05:59 PM 154,624 liblzma-f8552861.dll 04/27/2022 05:59 PM 193,536 libpng16-f2a3ac70.dll 04/27/2022 05:59 PM 227,840 libssh2-54993272.dll 04/27/2022 05:59 PM 1,281,024 libxml2-64eb6d5f.dll 04/27/2022 05:59 PM 253,659 LICENSES_bundled.txt 04/27/2022 05:59 PM 137,728 mfhdf-b1f9baf5.dll 04/27/2022 05:59 PM 565,640 msvcp140.dll 04/27/2022 05:59 PM multimethod 04/27/2022 05:59 PM multimethod-1.8.dist-info 04/27/2022 05:59 PM 1,092,608 netcdf-b8bee6dc.dll 04/27/2022 05:59 PM nlopt 04/27/2022 05:59 PM nlopt-2.7.1.dist-info 04/27/2022 05:59 PM nptyping 04/27/2022 05:59 PM nptyping-2.0.0.dist-info 04/27/2022 05:59 PM numpy 04/27/2022 05:59 PM numpy-1.22.3.dist-info 04/27/2022 05:59 PM 113,632,256 OCP.cp39-win_amd64.pyd 04/27/2022 05:59 PM ocp_vtk-7.5.3.0.dist-info 04/27/2022 05:59 PM 29,184 ogg-e38dc37e.dll 04/27/2022 05:59 PM pip 04/27/2022 05:59 PM pip-22.0.4.dist-info 04/27/2022 05:58 PM pkg_resources 04/27/2022 05:59 PM 2,999,808 proj_7_2-671605b1.dll 04/27/2022 05:59 PM pyparsing 04/27/2022 05:59 PM pyparsing-3.0.8.dist-info 04/27/2022 05:58 PM setuptools 04/27/2022 05:58 PM setuptools-49.2.1.dist-info 04/27/2022 05:59 PM 1,460,736 sqlite3-dd39700b.dll 04/27/2022 05:59 PM 287,744 tbb-17b02d2c.dll 04/27/2022 05:59 PM 131,584 tbbmalloc-24341326.dll 04/27/2022 05:59 PM tests 04/27/2022 05:59 PM 210,432 theora-ad1efaaf.dll 04/27/2022 05:59 PM 698,368 tiff-9eb9c421.dll 04/27/2022 05:59 PM 1,738,240 TKernel-d4900e62.dll 04/27/2022 05:59 PM 315,392 TKG2d-af7ff42f.dll 04/27/2022 05:59 PM 974,336 TKG3d-2f65fb9e.dll 04/27/2022 05:59 PM 1,787,392 TKMath-e14c04cc.dll 04/27/2022 05:59 PM typing_extensions-4.2.0.dist-info 04/27/2022 05:59 PM 70,613 typing_extensions.py 04/27/2022 05:59 PM typish 04/27/2022 05:59 PM typish-1.9.3.dist-info 04/27/2022 05:59 PM 932,352 vtkChartsCore-9.0-a07ade5b.dll 04/27/2022 05:59 PM 106,496 vtkCommonColor-9.0-87ee4902.dll 04/27/2022 05:59 PM 166,400 vtkCommonComputationalGeometry-9.0-4f06c132.dll 04/27/2022 05:59 PM 4,744,192 vtkCommonCore-9.0-4ec86a33.dll 04/27/2022 05:59 PM 3,540,480 vtkCommonDataModel-9.0-8d940f29.dll 04/27/2022 05:59 PM 774,656 vtkCommonExecutionModel-9.0-953aa4ba.dll 04/27/2022 05:59 PM 124,416 vtkCommonMath-9.0-899fa293.dll 04/27/2022 05:59 PM 117,760 vtkCommonMisc-9.0-b4b3de35.dll 04/27/2022 05:59 PM 25,600 vtkCommonPython-9.0-da6b908b.dll 04/27/2022 05:59 PM 95,744 vtkCommonSystem-9.0-1a27d27d.dll` |
|
@roipoussiere see above, installation said successful using the wheel, but then dlls are not found. |
|
There is also a user in issue #1068 who seems to be having the same issue while trying to install in Blender. |
|
Thanks for testing. I've stumbled on at least one issue on Windows which may explain the reported errors above. Looking through the build action logs, I see that only four TK*.dll files are included in the wheel: The listed TK*.dll files match the directory listing in the error report above. My understanding is that many more (or maybe all) TK*.dll files should have been included. The list of dll's to include is generated by delvewheel, which in turn uses pefile to parse imports. pefile imposes an upper limit on the number of imports to parse, causing the import list to be truncated. A simple fix to try would be to replace the two calls to @roipoussiere |
|
A new version of delvewheel (0.0.22) has been released with some fixes, so the workaround above is no longer needed. We can just trigger the build action to get new wheels and test again on Windows. @roipoussiere |
|
i'm happy to retest whenever we're ready |
|
@fpq473 @roipoussiere I'm trying to get a parallel CI setup going as a precursor to integrating this work into the main pipeline. I'm getting this encoding error, but only on Windows. Did either of your run into this? Also, because this method uses a conda environment to build the wheels, we can't trigger this build off of a push to the master branch as the changes will not be available in the Anaconda package repository yet. I'm considering putting this and one or two other builds in a separate repo designed for that purpose. |
|
I am unable to verify this in a Windows environment on Github Actions, but my theory is that the default encoding there is cp1252. So what may be happening is that we are trying to write a UTF-8 string to a cp1252-encoded file, causing the error. Demo: One fix to try would be to specify the encoding when opening the licenses file: with open("LICENSES_bundled.txt", "w", encoding="utf-8") as f:
# ^^^^^^^^^^^^^^in setup.py: https://github.com/jmwright/ocp-build-system/blob/741e6c579d43e3281420549923ea3e3c1a2f822e/setup.py#L199 You would have to remove There are other workarounds like defaulting to UTF-8, but I'm not so familiar with best practices on Windows: https://dev.to/methane/python-use-utf-8-mode-on-windows-212i |
|
@fpq473 Thanks for the tip. Setting the |
|
Here's a link to the new builds if anyone else wants to take a look: https://github.com/jmwright/ocp-build-system/actions/runs/2272471797 |
|
Thanks for building. I just tested on Windows with Python 3.9 and it looks fine (and the build is much faster, which is expected with the new version of delvewheel). Note that there is a test step in the build action, but I'm thinking that maybe it's running inside the conda env (which has OCP installed!) instead of starting from scratch in an empty environment. That's why the tests passed before here but we got the error reported above. I think that conda is being activated on L59 (see the discussion about |
|
Ok I will test this again tonight also |
|
@dcowden You'll want to use one of the install lines that I posted here for the Blender user to try. I think I have a method worked out to trigger a wheel build/release in Azure after the OCP uploads to the Anaconda package repo complete sucessfully, but I want to get all the pieces of the full integration with PyPi working before I create a PR on OCP's repo. |
|
@fpq473 I think I have the testing step fixed. Before, the test was checking what was already on PyPi, not the wheel that had just been generated locally. https://github.com/jmwright/ocp-build-system/actions/runs/2276238784 I have also confirmed that the Python 3.8 wheel works in MacOS 10.14.6. |
|
ok @jmwright @roipoussiere @fpq473 I tested. |
|
That's great news. I'm considering manually building a release on CadQuery's PyPi account so we can get people using these wheels, and then starting on the OCP integration after that. I think that doing the release manually this time will make it easier to request the size increase from PyPi for our packages anyway. |
|
In case you missed it, and in case there's a clash of names, @roipoussiere obtained a PyPI file limit increase to 200 MB for the ocp-vtk wheels he published. See #1048 (comment) above. |
|
That's one of the questions I have. The |
|
Maybe the package name should be |
|
@jmwright yes i agree, isnt it useful sometimes to be able to refer to packages with wildcards? cadquery-* |
|
Either works. It's probably a good idea to drop the "vtk" part of the name in case we figure out how to avoid bundling it, and include it as a dependency instead. |
|
I've updated the release to use the For example (Linux with Python 3.8): I'll work on getting these wheels pushed to PyPi next. |
For Debian11 with Python 3.9.2, all these work except nptyping package |
|
Thanks for pointing this out @hastelloy. I can replicate, assuming you had nptyping 1.4.4 already installed when you ran your pip command -- pip will not upgrade in this scenario. I am not able to replicate from a new venv. That said, this is a good reminder that we need to remember to specify the nptyping version when building cadquery wheels, as was done for conda. |
|
I'm going to close this issue since we now have OCP wheels. If anyone is still having issues with the wheels, please feel free to open a new issue to get help. The documentation here shows how to install CadQuery via pip and what the limitations are, and the OCP wheel can be installed by itself using |
|
Big thanks to @fpq473 for getting this going. |
I have found a way to create self-contained OCP wheels on Linux, allowing cadquery and all its dependencies to be installed via pip. I am sharing it here in the hopes that others may find it helpful, either for personal pip workflows or as a step towards official PyPI wheels.
I note that #153 (comment) requests a working pipeline, presumably Azure pipelines for all platforms. This is NOT that yet; this is only a way to build a wheel, and has only been tested on Linux.
The wheel is created from a conda environment with OCP installed. The OCP and vtk files in the python import path are blindly copied into a wheel. This wheel is of course unusable, but then we simply use auditwheel to bundle external libraries and produce a working manylinux_2_31_x86_64 wheel. This wheel is 160 MB, which is large but probably can be accommodated by PyPI.
A working cadquery venv can be created by pip installing this wheel along with
git+https://github.com/CadQuery/cadquery,ezdxf,ipython,multimethod,nlopt, andnptyping<2. Thanks to @roipoussiere who has also tested this.The code is available in this gist; see the docstring of
setup.pyfor details. Testers on macOS and Windows will have to replace the use of auditwheel with delocate and delvewheel respectively. The created wheels may not be distributed until the requisite license files have been added into the wheel.The text was updated successfully, but these errors were encountered: