[Python] Uploading nightly wheels to the scientific python nightly channel

[jarrodmillman]

Describe the enhancement requested

It looks like you already build nightly Python wheels:

• https://github.com/apache/arrow/blob/main/docs/source/developers/python.rst#installing-nightly-packages

Would you be interested/willing to upload nightly wheels to he scientific-python nightly channel instead:

• https://anaconda.org/scientific-python-nightly-wheels/

This would make the wheels easier to discover by hosting them in a "standard" place. It will also make it easier for downstream projects to test against multiple packages, since they won't need to specify multiple locations in their CI scripts.

You can read more about why and how we are building / testing against nightly wheels here:
https://scientific-python.org/specs/spec-0004/

We also provide a GitHub action to upload and manage nightly builds:
https://github.com/scientific-python/upload-nightly-action/

We @matthewfeickert, @larsoner, @tupui are happy to help.

Component(s)

Python

Linked PRs

• GH-40216: [CI][Packaging][Python] Upload pyarrow nightly wheels to scientific python channel on Anaconda #43862
• GH-40216: [Python][CI][Packaging] Upload nightly wheels to main label of scientific-python-nightly-wheels channel #43932
• GH-40216: [Python][CI][Packaging] Don't upload sdist to scientific-python nightly channel (only wheels) #43943

[jorisvandenbossche]

cc @raulcd

This sounds as an interesting idea to me. My understanding is that our nightly wheels are not "official" release artifacts anyway, so it shouldn't matter too much where it is hosted? (also not sure if the current hosting location is actually tied to the Apache foundation)

I suppose the main cost is the cost of switching for current users of this (and for them to first find out that the old link no longer gives them the latest packages, at the time we would decide to stop uploading there as well)

[matthewfeickert]

👋 @jorisvandenbossche Just wanted to see if the team had any further thoughts on this. Let us know if we can help answer any questions.

[raulcd]

We probably could upload wheels to both places. In terms of discoverability we have to advise to users that the wheels provided are not official releases and should be used at their own risk. Similar to what we have at the documentation:
These packages are not official releases. Use them at your own risk.
This is because the Apache Release Policy (https://www.apache.org/legal/release-policy.html#what) says:

During the process of developing software and preparing a release, various packages are made available to the developer community for testing purposes. Do not include any links on the project website that might encourage non-developers to download and use nightly builds, snapshots, release candidates, or any other similar package. The only people who are supposed to know about such packages are the people following the dev list (or searching its archives) and thus aware of the conditions placed on the package. If you find that the general public are downloading such test packages, then remove them.

I think we are ok publishing the nightly wheels under the scientific python nightly channel.

[matthewfeickert]

I think we are ok publishing the nightly wheels under the scientific python nightly channel.

@raulcd Great! To move forward we need someone in the Arrow organization to do the following:

• Create an account on Anaconda Cloud
• Reply here with the link to your account (https://anaconda.org/ktou/)
• Make a new Anaconda Cloud API token with permissions described in https://scientific-python.org/specs/spec-0004/#process-for-adding-new-projects

• Save the API token as a repo GitHub Actions secret named ANACONDA_ORG_UPLOAD_TOKEN
• Direct me (@matthewfeickert) to a pyarrow nightly wheel for me to upload to create the package in the registry
• Verify that I have added you to the pyarrow team "Group" on https://anaconda.org/scientific-python-nightly-wheels (I've already added @jorisvandenbossche)
• Verify that I have added the package scientific-python-nightly-wheels/pyarrow to the pyarrow team "Group"

If you'd also like to use the scientific/upload-nightly-action GitHub Action please check out the action repo's README. If you need to collect wheels from different CI jobs, consider creating a new GitHub Actions workflow with logic similar to matplotlib's or Awkward's.

[kou]

Could you add me https://anaconda.org/ktou/ to the pyarrow team "Group" too?

[matthewfeickert]

@kou done. As soon as you generate an API token you can start uploading to https://anaconda.org/scientific-python-nightly-wheels/pyarrow. 👍 Let's leave this Issue open though until the first automated CI/CD upload passes.

[kou]

Thanks! I confirmed that I'm a member of the group.

@assignUser We need to ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN secret to our GitHub Actions. Could you also join the group, generate your API token and ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN?

[matthewfeickert]

Could you also join the group, generate your API token and ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN?

@assignUser Can you also share your Anaconda Cloud account name so I can add you to the pyarrow group?

[assignUser]

@matthewfeickert Sorry I missed the notifications 🙈 https://anaconda.org/assignuser

[matthewfeickert]

@assignUser added. 👍

[assignUser]

I'll take care of adding the token with infra

[assignUser]

See https://issues.apache.org/jira/browse/INFRA-25697

[assignUser]

@kou infra just added the secret so we can create a workflow (will close the ticket once we can confirm it works)

[kou]

Thanks!

[assignUser]

INFRA will add the secret but with a ARROW_ prefix for organizational reasons, as we have to pull it into an envar anyway it shouldn't matter.

[matthewfeickert]

INFRA will add the secret but with a ARROW_ prefix for organizational reasons

That's totally fine. The secret name is arbitrary even if the GitHub Actions uploaded is used, but I just suggest a name to keep it simple for people. 👍

[kou]

OK. What is the next action then? We need to update our workflows?

[assignUser]

@kou As far as I understand we need to create a workflow similiar to r_nightly.yml that collects the pyarrow wheels from crossbow and then uploads them to the anaconda channel using the new secret.

[matthewfeickert]

we need to create a workflow similar to r_nightly.yml that collects the pyarrow wheels from crossbow and then uploads them to the anaconda channel using the new secret.

Yeah. #40216 (comment) has some linked examples of how other projects are doing this.

[kou]

I see! Thanks!

[matthewfeickert]

👋 @kou @assignUser I just wanted to follow up on this to see if there are any questions on your end RE: getting deployments up.

[kou]

@assignUser Can you work on this? Or should someone work on this?

[assignUser]

@kou I won't get to it quickly but I can take a look 'soon'(tm), maybe @matthewfeickert would be interested in working on it? The way we generate the wheels is a bit complex but it seems you have looked into that already?

[matthewfeickert]

@assignUser Realistically I don't have time at the moment, given that summer is peak everything time for academia.

[jorisvandenbossche]

As far as I understand we need to create a workflow similiar to r_nightly.yml that collects the pyarrow wheels from crossbow and then uploads them to the anaconda channel using the new secret.

For uploading the wheels, we currently have the github_upload_gemfury github actions macro, that is used to upload the built nightly wheels directly from the jobs building and testing them, for example at

arrow/dev/tasks/python-wheels/github.linux.yml

Line 108 in 712cfe6

{{ macros.github_upload_gemfury("arrow/python/repaired_wheels/*.whl")|indent }}

We could do this with a similar approach adding an extra macro to upload the wheels to anaconda.org?
Or @assignUser is there a reason for (or advantage to) using an approach like r_nightly.yml collecting all artifacts and uploading from a run on this repo? (the file mention "Due to authorization requirements, this upload can't be done from the crossbow repository.", but is that because this one is syncing with nightlies.apache.org? Uploading to gemfury doesn't seem to have that issue)

[raulcd]

Some of the nightly wheels have also failed to upload similar to the failures that we found on the latest PR from @jorisvandenbossche :
wheel-macos-monterey-cp310-amd64
wheel-windows-cp310-amd64
wheel-windows-cp313-amd64
wheel-windows-cp39-amd64
Those errors might have been temporary network errors but we should monitor and maybe add some retry mechanism to the upload if this is common.

[matthewfeickert]

Quick comments as I'm in transit (sorry):

• yes, if you want the dev label please add that in addition to main.

c.f. https://github.com/scientific-python/upload-nightly-action/blob/b3d150d89b9500cd5623f211d6214f1d8b50fac5/action.yml#L21-L24 and https://github.com/scientific-python/upload-nightly-action/blob/b3d150d89b9500cd5623f211d6214f1d8b50fac5/cmd.sh#L44-L53

• In general we'd prefer to only upload wheels so that people using the index for testing don't accidentally run into situations in which they trigger a build from sdist.

[jorisvandenbossche]

• In general we'd prefer to only upload wheels so that people using the index for testing don't accidentally run into situations in which they trigger a build from sdist.

Opened a PR to remove the sdist uploading: #43943

[raulcd]

Issue resolved by pull request 43943
#43943

[raulcd]

With the latest changes I've closed the issue as done. Please re-open or open a new issue if any follow up is necessary.
Thanks @jorisvandenbossche for following this up!

[larsoner]

There are no longer any Windows wheels up at https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/pyarrow/ , is this intentional? At least I think there used to be Windows wheels

[assignUser]

Looks like the windows nightly builds have failed for a few days and the channel is removing old files?

[larsoner]

Yeah I know there is some removal policy, I thought it was a month but I could be wrong (and it might also be per-project configurable, not sure!). A few days seems like too short a removal period in any case.

[matthewfeickert]

Yeah I know there is some removal policy, I thought it was a month but I could be wrong

It is 30 days (c.f. https://github.com/scientific-python/upload-nightly-action/blob/07ba11e3fcd43a475f9f56a9b06555aa2c621136/.github/workflows/remove-wheels.yml#L61-L66)

(and it might also be per-project configurable, not sure!).

Currently, no. There is a "never delete anything" option, but the only project that had a good use case for that no longer needs it.

[raulcd]

From my understanding there is still a max number of wheels allowed too, right? Taking into account all our different wheels (macos arm + x86_64, manylinux_2014 and manylinux_2_28 both for arm + x86_64 for Python versions from 3.9 to 3.13) we get about ~6 days of wheels. See the oldest ones here: https://anaconda.org/scientific-python-nightly-wheels/pyarrow/files?page=4
The Windows nightly jobs have failed for slightly longer so they've just been removed.

[matthewfeickert]

From my understanding there is still a max number of wheels allowed too, right?

We don't impose a maximum number of artifacts. If any project takes up too much total storage we would look into how to reduce it, but the automatic removal workflow is meant to avoid this and also have "nightly" mean something relatively recent.

Taking into account all our different wheels (macos arm + x86_64, manylinux_2014 and manylinux_2_28 both for arm + x86_64 for Python versions from 3.9 to 3.13) we get about ~6 days of wheels.

I assume (don't have time to check) you're just seeing that we also keep only the last 5 uploads by default:

https://github.com/scientific-python/upload-nightly-action/blob/07ba11e3fcd43a475f9f56a9b06555aa2c621136/.github/workflows/remove-wheels.yml#L19

again, focusing on "nightly".

[kou]

Our nightly Windows wheel builds are broken recently. (They will be fixed by #45442.)

Is it related?

[raulcd]

I assume (don't have time to check) you're just seeing that we also keep only the last 5 uploads by default:

If I understand correctly that snippet on the GH actions if we upload a single wheel with a newer version, all wheels that have an older than 5 versions will be removed. That seems to match what we are seeing, we are still uploading newer wheels but the Windows wheels stopped to be uploaded 15 days ago due to CI failures.

Our nightly Windows wheel builds are broken recently. (They will be fixed by #45442.) Is it related?

Yes, it will be fixed today but it's more to understand when are those removed as this is the second time that something like this happens.

[matthewfeickert]

If I understand correctly that snippet on the GH actions if we upload a single wheel with a newer version, all wheels that have an older than 5 versions will be removed.

@raulcd Yes. Within 24 hours of an upload the CI will trim it down to 5 versions max (so you can have more than up there for a brief period).

I see that PR #45442 has been merged now (👍). Let us know if in the next automated upload the Windows wheels aren't restored.