-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build modernization (GHA, wheels, setuptools) #407
Merged
Merged
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit
Hold shift + click to select a range
0ba72f8
Script for build manylinux wheels
bsolomon1124 cfb1d7d
Build libyaml ourselves, nix yum install
bsolomon1124 4327345
Build manylinux2014 wheels in CI
bsolomon1124 99568e5
chmod +x
bsolomon1124 cc9f531
Typo
bsolomon1124 a7de8d2
Remove deprecated 'sudo' arg
bsolomon1124 79bbe26
Use --workdir to put ourselves in /io
bsolomon1124 d2139d0
Build pure-Python wheels
bsolomon1124 8d9be9e
Typo on pip flag
bsolomon1124 dcb137a
Remove sudo, check UID 0 instead
bsolomon1124 3165b50
Shell syntax
bsolomon1124 2a2bfc6
Avoid error where we may be root but not have sudo itself available
bsolomon1124 115228c
ls wheel dir after successful build
bsolomon1124 59bcd38
Namespace _yaml extension under yaml package
bsolomon1124 75ccf4c
[snapshot] Simplify setup.py per 'Distributing Cython modules' docume…
bsolomon1124 3ad7bb0
Make build_ext happy with package structure
bsolomon1124 3108af5
Specify USE_CYTHON for build
bsolomon1124 9803591
Fix ternary
bsolomon1124 dee7252
Revert "Fix ternary"
bsolomon1124 57cb517
Revert "Specify USE_CYTHON for build"
bsolomon1124 76d7f19
Revert "Make build_ext happy with package structure"
bsolomon1124 f69087c
Revert "[snapshot] Simplify setup.py per 'Distributing Cython modules…
bsolomon1124 46eed93
Reset to 59bcd38 / cherry-pick 3ad7bb0
bsolomon1124 c0180aa
Update extension path
bsolomon1124 d6f7f87
Update ignored build artifacts
bsolomon1124 cb71ba5
Build MacOS wheels
bsolomon1124 c2203e7
Create a 'stub' _yaml package for relocated _yaml extension
bsolomon1124 6269678
Incorporate macos into Travis build matrix
bsolomon1124 8e9cebe
Put _yaml stub under lib/ to aggree with package_dir
bsolomon1124 393985f
Use --disable-dependency-tracking for one-time build (see https://git…
bsolomon1124 ee1dc2b
No ldconfig on MacOS
bsolomon1124 b509351
Don't use -m flag when not in virtual env
bsolomon1124 54abbdc
Downgrade to 2.7.17 (.18 not avail via pyenv-build yet?)
bsolomon1124 02de749
Echo available Python versions if pyenv cannot find
bsolomon1124 6ecc19d
Remove verbose flag when removing temp dir
bsolomon1124 4f9aae0
Convert wheel build from Travis CI to GitHub Actions
bsolomon1124 71fe365
Fixups for GitHub Actions build environment
bsolomon1124 c781067
Put brew cmds in the right place
bsolomon1124 f6225d2
Fix brew flag
bsolomon1124 510d297
CI runner failures ... who knows
bsolomon1124 27a27ce
Apparently -q is not actually a global option...
bsolomon1124 6656bd7
libyaml build/test also wants tac from coreutils
bsolomon1124 c577b2f
Pure-Py wheels need --without-libyaml
bsolomon1124 007869b
Attempt manylinux1_x86_64 over manylinux2014
bsolomon1124 9dba870
Build across manylinux1 + manylinux2014
bsolomon1124 473a885
CI to GHA, wheel builds
nitzmahone 6a292fa
PEP517, setuptools conversion
nitzmahone 6c9faab
Windows tweaks
nitzmahone File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
CI to GHA, wheel builds
* manylinux, macos
- Loading branch information
commit 473a885e0bb1422705bf4c5ab9e21f3572b2d83d
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,256 @@ | ||
--- | ||
|
||
name: PyYAML CI | ||
on: | ||
push: | ||
pull_request: | ||
workflow_dispatch: | ||
inputs: | ||
build_wheels: | ||
default: false | ||
env: | ||
LIBYAML_REPO: https://github.com/yaml/libyaml | ||
LIBYAML_REF: '0.2.5' | ||
jobs: | ||
python_sdist: | ||
name: pyyaml sdist | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: checkout pyyaml | ||
uses: actions/checkout@v2 | ||
|
||
- name: install a python | ||
uses: actions/setup-python@v2 | ||
with: | ||
python-version: 3.x | ||
|
||
- name: install build deps | ||
run: | | ||
python -V | ||
|
||
python -m pip install Cython setuptools wheel | ||
|
||
- name: build sdist | ||
run: | | ||
export PYYAML_FORCE_CYTHON=1 | ||
export PYYAML_FORCE_LIBYAML=1 | ||
|
||
# FIXME: move this to `build` once our build is PEP517 OK | ||
python setup.py sdist | ||
|
||
# ensure exactly one artifact | ||
shopt -s nullglob | ||
DISTFILES=(dist/*.tar.gz) | ||
if [[ ${DISTFILES[@]} -ne 1 ]]; then | ||
echo "unexpected content in dist dir: $(ls dist/*.tar.gz)" | ||
exit 1 | ||
fi | ||
|
||
- name: test sdist | ||
run: | | ||
# install some libyaml headers | ||
# TODO: should we smoke test the sdist against the libyaml we built? | ||
sudo apt update | ||
sudo apt install libyaml-dev -y | ||
|
||
# remove Cython to ensure we use only what's in the sdist | ||
python -m pip uninstall Cython -y | ||
|
||
# pass no extra args- we should auto-install with libyaml since it's present | ||
python -m pip install dist/*.tar.gz -v | ||
|
||
python packaging/build/smoketest.py | ||
|
||
- name: upload sdist artifact | ||
uses: actions/upload-artifact@v2 | ||
with: | ||
name: dist | ||
path: dist/*.tar.gz | ||
|
||
|
||
linux_libyaml: | ||
name: libyaml ${{ matrix.arch }} ${{ matrix.platform }} | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
platform: | ||
# manylinux1 is forward-compatible to 2010/2014 | ||
#- manylinux2014 | ||
#- manylinux2010 | ||
- manylinux1 | ||
arch: | ||
- x86_64 | ||
env: | ||
DOCKER_IMAGE: quay.io/pypa/${{ matrix.platform }}_${{ matrix.arch }} | ||
steps: | ||
- name: check cached libyaml state | ||
id: cached_libyaml | ||
uses: actions/cache@v2 | ||
with: | ||
path: | | ||
libyaml | ||
key: libyaml_${{ matrix.platform }}_${{ matrix.arch }}_${{ env.LIBYAML_REF }} | ||
|
||
- name: checkout pyyaml | ||
uses: actions/checkout@v2 | ||
if: steps.cached_libyaml.outputs.cache-hit != 'true' | ||
|
||
- name: build libyaml | ||
run: | | ||
docker run --rm -v $(pwd):/io -e LIBYAML_REF -e LIBYAML_REPO --workdir /io "$DOCKER_IMAGE" /io/packaging/build/libyaml.sh | ||
if: steps.cached_libyaml.outputs.cache-hit != 'true' | ||
|
||
linux_pyyaml: | ||
needs: linux_libyaml | ||
name: pyyaml ${{ matrix.arch }} ${{ matrix.platform }} ${{ matrix.python_tag }} | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
platform: | ||
# so long as manylinux1 container builds work, they're forward-compatible to 2010/2014 | ||
# - manylinux2014 | ||
# - manylinux2010 | ||
- manylinux1 | ||
arch: | ||
- x86_64 | ||
python_tag: | ||
# NB: manylinux >=2014 containers don't have Python 2.7, so we have to use exclude to skip it | ||
- cp27-cp27mu | ||
- cp36-cp36m | ||
- cp37-cp37m | ||
- cp38-cp38 | ||
- cp39-cp39 | ||
# exclude: | ||
# - platform: manylinux2014 | ||
# arch: x86_64 | ||
# python_tag: cp27-cp27mu | ||
env: | ||
AW_PLAT: ${{ matrix.platform }}_${{ matrix.arch }} | ||
DOCKER_IMAGE: quay.io/pypa/${{ matrix.platform }}_${{ matrix.arch }} | ||
PYTHON_TAG: ${{ matrix.python_tag }} | ||
PYYAML_BUILD_WHEELS: 1 | ||
steps: | ||
- uses: actions/checkout@v2 | ||
|
||
- name: fetch cached libyaml | ||
id: cached_libyaml | ||
uses: actions/cache@v2 | ||
with: | ||
path: | | ||
libyaml | ||
key: libyaml_${{ matrix.platform }}_${{ matrix.arch }}_${{ env.LIBYAML_REF }} | ||
|
||
- name: ensure libyaml fetched | ||
run: exit 1 | ||
if: steps.cached_libyaml.outputs.cache-hit != 'true' | ||
|
||
- name: start container | ||
run: | | ||
docker run --name worker -t -d --rm -v $(pwd):/io "$DOCKER_IMAGE" bash | ||
|
||
- name: build/test/package | ||
run: | | ||
docker exec -e PYTHON_TAG -e PYYAML_RUN_TESTS -e PYYAML_BUILD_WHEELS -e AW_PLAT --workdir /io worker \ | ||
/io/packaging/build/manylinux.sh | ||
|
||
# ensure exactly one artifact | ||
shopt -s nullglob | ||
DISTFILES=(dist/*.whl) | ||
if [[ ${#DISTFILES[@]} -ne 1 ]]; then | ||
echo -e "unexpected dist content:\n\n$(ls)" | ||
exit 1 | ||
fi | ||
|
||
- uses: actions/upload-artifact@v2 | ||
with: | ||
#name: wheel_${{ matrix.platform }}_${{ matrix.arch }}_${{ matrix.python_tag }} | ||
name: dist | ||
path: dist/*.whl | ||
|
||
macos_libyaml: | ||
name: libyaml ${{ matrix.arch }} ${{ matrix.platform }} | ||
runs-on: ${{ matrix.platform }} | ||
strategy: | ||
matrix: | ||
platform: | ||
- macos-10.15 | ||
arch: | ||
- x86_64 | ||
steps: | ||
- name: check cached libyaml state | ||
id: cached_libyaml | ||
uses: actions/cache@v2 | ||
with: | ||
path: | | ||
libyaml | ||
key: libyaml_${{ matrix.platform }}_${{ matrix.arch }}_${{ env.LIBYAML_REF }} | ||
|
||
- name: checkout pyyaml | ||
uses: actions/checkout@v2 | ||
if: steps.cached_libyaml.outputs.cache-hit != 'true' | ||
|
||
- name: build libyaml | ||
env: | ||
MACOSX_DEPLOYMENT_TARGET: '10.9' | ||
run: | | ||
brew install automake coreutils | ||
bash ./packaging/build/libyaml.sh | ||
if: steps.cached_libyaml.outputs.cache-hit != 'true' | ||
|
||
|
||
macos_pyyaml: | ||
needs: macos_libyaml | ||
name: pyyaml ${{ matrix.arch }} ${{ matrix.platform }} ${{ matrix.python_tag }} | ||
runs-on: ${{ matrix.platform }} | ||
strategy: | ||
matrix: | ||
platform: | ||
- macos-10.15 | ||
arch: | ||
- x86_64 | ||
python_tag: | ||
- cp27* | ||
- cp36* | ||
- cp37* | ||
- cp38* | ||
- cp39* | ||
steps: | ||
- name: checkout pyyaml | ||
uses: actions/checkout@v2 | ||
|
||
- name: get cached libyaml state | ||
id: cached_libyaml | ||
uses: actions/cache@v2 | ||
with: | ||
path: | | ||
libyaml | ||
key: libyaml_${{ matrix.platform }}_${{ matrix.arch }}_${{ env.LIBYAML_REF }} | ||
|
||
- name: ensure libyaml fetched | ||
run: exit 1 | ||
if: steps.cached_libyaml.outputs.cache-hit != 'true' | ||
|
||
- name: install a python | ||
uses: actions/setup-python@v2 | ||
with: | ||
python-version: 3.x | ||
|
||
- name: build/test/package | ||
env: | ||
CIBW_BUILD: ${{ matrix.python_tag }} | ||
CIBW_BUILD_VERBOSITY: 1 | ||
run: | | ||
bash ./packaging/build/macos.sh | ||
|
||
# ensure exactly one artifact | ||
shopt -s nullglob | ||
DISTFILES=(dist/*.whl) | ||
if [[ ${#DISTFILES[@]} -ne 1 ]]; then | ||
echo -e "unexpected dist content:\n\n$(ls)" | ||
exit 1 | ||
fi | ||
|
||
- uses: actions/upload-artifact@v2 | ||
with: | ||
name: dist | ||
path: dist/*.whl |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,10 @@ | ||
include README LICENSE CHANGES setup.py | ||
include CHANGES README LICENSE Makefile pyproject.toml setup.py | ||
recursive-include lib/yaml *.py | ||
recursive-include lib/_yaml *.py | ||
recursive-include lib3/yaml *.py | ||
recursive-include lib3/_yaml *.py | ||
recursive-include examples *.py *.cfg *.yaml | ||
recursive-include tests/data * | ||
recursive-include tests/lib *.py | ||
recursive-include tests/lib3 *.py | ||
recursive-include yaml/ * |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nitzmahone why only "mu"? py2 has two separate builds to link against
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You also need "cp27-cp27m" to cover that.