-
Notifications
You must be signed in to change notification settings - Fork 14.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Limit Apache Beam's numpy used #46286
Conversation
Separating beam out reveald that the lower limit for numpy migh lead to installation problems on newer package managers and python versions (lack of distutils installed by default in Python 3.12 in causes older versions of numpy to not build and uv / pip trying to install it will fail. This PR lower-binds numpy to mid-2023 version.
Example failure: https://github.com/apache/airflow/actions/runs/13047585952/job/36404137972#step:6:5204 Fails when tries to build 1.24.4 on 3.12 Resolved 178 packages in 4.30s
× Failed to build `numpy==1.24.4`
├─▶ The build backend returned an error
╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit
status: 1)
[stderr]
Traceback (most recent call last):
File "<string>", line 8, in <module>
File
"/root/.cache/uv/builds-v0/.tmppPFSGG/lib/python3.12/site-packages/setuptools/__init__.py",
line 10, in <module>
import distutils.core
ModuleNotFoundError: No module named 'distutils'
hint: `distutils` was removed from the standard library in Python 3.12.
Consider adding a constraint (like `numpy >1.24.4`) to avoid building a
version of `numpy` that depends on `distutils`.
help: `numpy` (v1.24.4) was included because `apache-airflow[apache-beam]`
(v3.0.0.dev0) depends on `pyarrow` (v14.0.1) which depends on
`numpy` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So it wasnt flakiness after all. Yeah this workaround sounds OK for now!
It's a good long term one actually- because we prevent installing old versions that are not installable on newer python versions :) |
Merging to get next canary builds to run it |
Separating beam out reveald that the lower limit for numpy migh lead to installation problems on newer package managers and python versions (lack of distutils installed by default in Python 3.12 in causes older versions of numpy to not build and uv / pip trying to install it will fail. This PR lower-binds numpy to mid-2023 version.
Cool! Yeah, did not notice this but also in example DAGs numpy < 1.26 failed with Python 3.12. Good that this is also fixed here! |
Separating beam out reveald that the lower limit for numpy migh lead to installation problems on newer package managers and python versions (lack of distutils installed by default in Python 3.12 in causes older versions of numpy to not build and uv / pip trying to install it will fail. This PR lower-binds numpy to mid-2023 version.
Separating beam out reveald that the lower limit for numpy migh lead to installation problems on newer package managers and python versions (lack of distutils installed by default in Python 3.12 in causes older versions of numpy to not build and uv / pip trying to install it will fail. This PR lower-binds numpy to mid-2023 version.
Separating beam out reveald that the lower limit for numpy migh lead to installation problems on newer package managers and python versions (lack of distutils installed by default in Python 3.12 in causes older versions of numpy to not build and uv / pip trying to install it will fail.
This PR lower-binds numpy to mid-2023 version.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.