Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-40216: [CI][Packaging][Python] Upload pyarrow nightly wheels to scientific python channel on Anaconda #43862

Merged
merged 16 commits into from
Sep 2, 2024
Merged

GH-40216: [CI][Packaging][Python] Upload pyarrow nightly wheels to scientific python channel on Anaconda #43862

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a690ef4
GH-40216: [CI][Packaging][Python] Upload pyarrow nightly wheels to sc…
raulcd Aug 28, 2024
bc682b1
Temporarily remove check for default branch to test upload
raulcd Aug 28, 2024
e5f45ef
Try manual upload to anaconda
raulcd Aug 28, 2024
5336576
Add env for token
raulcd Aug 28, 2024
64a9de6
Fix yaml for GitHub syntax instead of Azure :)
raulcd Aug 28, 2024
27a7109
Install conda before uploading wheel
raulcd Aug 28, 2024
3aeef04
Use setup-miniconda action
raulcd Aug 28, 2024
641edf6
Fix activate command
raulcd Aug 28, 2024
94d2e9d
Use the existing environment created by action instead of creating ne…
raulcd Aug 28, 2024
7354834
Try with setup-micromamba instead of setup-miniconda
raulcd Aug 28, 2024
3d96e6e
Fix identation
raulcd Aug 28, 2024
b6502bc
Use pip instead of conda for anaconda-client
raulcd Aug 29, 2024
14ea848
Use newer version not available on pip
raulcd Aug 29, 2024
26d7301
Add upload to scientific python to the rest of the wheels and sdist
raulcd Aug 29, 2024
8662076
Add sdist to wheels group on archery tasks
raulcd Aug 29, 2024
6ecb968
Add back guard to only upload if branch is main
raulcd Aug 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions dev/tasks/macros.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,18 @@ env:
{% endif %}
{% endmacro %}

{%- macro github_upload_wheel_scientific_python(pattern) -%}
{%- if arrow.is_default_branch() -%}
- name: Upload wheel to Anaconda scientific-python
shell: bash
run: |
python3 -m pip install git+https://github.com/Anaconda-Platform/[email protected]
anaconda -t ${CROSSBOW_SCIENTIFIC_PYTHON_UPLOAD_TOKEN} upload --force -u scientific-python-nightly-wheels --label dev {{ pattern }}
env:
CROSSBOW_SCIENTIFIC_PYTHON_UPLOAD_TOKEN: {{ '${{ secrets.CROSSBOW_SCIENTIFIC_PYTHON_UPLOAD_TOKEN }}' }}
{% endif %}
{% endmacro %}

{%- macro azure_checkout_arrow() -%}
- script: |
git clone --no-checkout --branch {{ arrow.branch }} {{ arrow.remote }} arrow
Expand Down
1 change: 1 addition & 0 deletions dev/tasks/python-sdist/github.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,4 @@ jobs:

{{ macros.github_upload_releases("arrow/python/dist/*.tar.gz")|indent }}
{{ macros.github_upload_gemfury("arrow/python/dist/*.tar.gz")|indent }}
{{ macros.github_upload_wheel_scientific_python("arrow/python/dist/*.tar.gz")|indent }}
1 change: 1 addition & 0 deletions dev/tasks/python-wheels/github.linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ jobs:

{{ macros.github_upload_releases("arrow/python/repaired_wheels/*.whl")|indent }}
{{ macros.github_upload_gemfury("arrow/python/repaired_wheels/*.whl")|indent }}
{{ macros.github_upload_wheel_scientific_python("arrow/python/repaired_wheels/*.whl")|indent }}

{% if arrow.is_default_branch() %}
- name: Push Docker Image
Expand Down
1 change: 1 addition & 0 deletions dev/tasks/python-wheels/github.osx.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,4 @@ jobs:

{{ macros.github_upload_releases("arrow/python/repaired_wheels/*.whl")|indent }}
{{ macros.github_upload_gemfury("arrow/python/repaired_wheels/*.whl")|indent }}
{{ macros.github_upload_wheel_scientific_python("arrow/python/repaired_wheels/*.whl")|indent }}
1 change: 1 addition & 0 deletions dev/tasks/python-wheels/github.windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ jobs:

{{ macros.github_upload_releases("arrow/python/dist/*.whl")|indent }}
{{ macros.github_upload_gemfury("arrow/python/dist/*.whl")|indent }}
{{ macros.github_upload_wheel_scientific_python("arrow/python/dist/*.whl")|indent }}

{% if arrow.is_default_branch() %}
- name: Push Docker Image
Expand Down
1 change: 1 addition & 0 deletions dev/tasks/tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ groups:

wheel:
- wheel-*
- python-sdist
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the sdist as part of the wheels job as several times we want to also trigger it when we trigger the wheels group but happy to remove it from here if we want to have it separate


linux:
- almalinux-*
Expand Down
Loading