Provide wheels for Windows

[filips123]

You should provide wheels for Wiindows and macOS. You can build them using CI system like Travis CI or GitHub Actions.

You should also provide source code on PyPI to allow users who don't use system with provided wheels to build package themselves.

[filips123]

I'm trying to switch building and testing to GitHub Actions and use cibuildwheels to build wheels for all systems.

The code is available in my builds-for-other-systems branch. Configuration for Python CI is in .github/workflows/python.yaml and building of libraries is in bindings/python/tools/prepare_build_environment.sh.

I was able to successfully build and test C++ API, but I have some problems with Python bindings:

• Action for publishing packages to PyPI (pypa/gh-action-pypi-publish@master) only works for Linux builds. That's why I have to use artifacts storage to save wheels when building on different systems and then get them back on Linux system to publish them to PyPI. However, because of other problems mentioned bellow, I wasn't able to test if this works.
  Because I'm always using the same artifact name, it may (actually probably is) be overwritten at every build. If so, artifact names should be different for each system build, but this can be fixed quite easily.
  See How to build and publish platform-specific wheels to PyPI pypa/gh-action-pypi-publish#15 for more details.

• I also have to use my own fork of cibuildwheel, because currently it doesn't provide a way to build Python package from a subdirectory of a project. Once Building wheels for package in subdirectory of project pypa/cibuildwheel#294 is implemented in some way, I can update to use real cibuildwheel source.

• Another problem is that PyPy can be supported, because pybind11 has some problems with it. But based on "'pybind11/pybind11.h' file not found" with PyPy (and does PyPy 5.7 exist?) pybind/pybind11#596, this isn't actually pybind11's problem but problem with pip on PyPy, so it can't be fixed easily.

• Building (at least tests as I'm not able to actually manually test wheels) on macOS works for some Python versions but fails on Python 3.8 because of "ImportError: dlopen(/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpfhmsx1rn/lib/python3.8/site-packages/pyonmttok.cpython-38-darwin.so, 2): Library not loaded: @rpath/libOpenNMTTokenizer.dylib". Output is available here.

• Building on Windows fails because make for ICU librariy relies on Makefile's SHELL variable, which on Windows is C:/Program Files/Git/usr/bin/sh.exe, so it contains a space and fails when make commands triy to run it. I tried to fix this with overwriting SHELL, but this didn't work. Output is available here.

• Building for Linux works and wheels are available here.

If someone has more experience with this, it would be nice to help me. Once this is fixed, I will cleanup commits and create PR, and a similar way of building could then also be used for CTranslate2.

[filips123]

/cc @guillaumekln @jsenellart

[guillaumekln]

Thanks for the work! That's very much appreciated.

However, I have no experience in building Python packages on macOS or Windows so I would need some time to process this. I have just a comment regarding the Windows ICU build failure: maybe you could just grab a binary release of ICU for this OS?

I know that the project TensorFlow Addons recently switched to GitHub Actions and is building wheels for the 3 systems. Maybe that could help you? https://github.com/tensorflow/addons/tree/master/.github/workflows

[jasperchua99]

@guillaumekln Possible to release one on aarch64? im using a jetson nano for this and it does not have a build

[guillaumekln]

@jasperchua99 Looks like manylinux2014 includes a aarch64 image, so this seems doable. Related links:

• Docker image: https://github.com/pypa/manylinux#manylinux2014

[guillaumekln]

The PR above is adding Python wheels for Windows. The tests are passing, but it would be helpful if someone can test a wheel on their Windows system.

You can download the wheels from the build page. Click on "Artifacts", download "python-wheels", and decompress the archive. You can then install the wheel matching your Python version.