Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit Apache Beam's numpy used #46286

Merged
merged 1 commit into from
Jan 30, 2025
Merged

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Jan 30, 2025

Separating beam out reveald that the lower limit for numpy migh lead to installation problems on newer package managers and python versions (lack of distutils installed by default in Python 3.12 in causes older versions of numpy to not build and uv / pip trying to install it will fail.

This PR lower-binds numpy to mid-2023 version.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Separating beam out reveald that the lower limit for numpy migh
lead to installation problems on newer package managers and python
versions (lack of distutils installed by default in Python 3.12 in
causes older versions of numpy to not build and uv / pip trying to
install it will fail.

This PR lower-binds numpy to mid-2023 version.
@potiuk
Copy link
Member Author

potiuk commented Jan 30, 2025

Example failure:

https://github.com/apache/airflow/actions/runs/13047585952/job/36404137972#step:6:5204

Fails when tries to build 1.24.4 on 3.12

Resolved 178 packages in 4.30s
    × Failed to build `numpy==1.24.4`
    ├─▶ The build backend returned an error
    ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit
        status: 1)
  
        [stderr]
        Traceback (most recent call last):
          File "<string>", line 8, in <module>
          File
        "/root/.cache/uv/builds-v0/.tmppPFSGG/lib/python3.12/site-packages/setuptools/__init__.py",
        line 10, in <module>
            import distutils.core
        ModuleNotFoundError: No module named 'distutils'
  
        hint: `distutils` was removed from the standard library in Python 3.12.
        Consider adding a constraint (like `numpy >1.24.4`) to avoid building a
        version of `numpy` that depends on `distutils`.
    help: `numpy` (v1.24.4) was included because `apache-airflow[apache-beam]`
          (v3.0.0.dev0) depends on `pyarrow` (v14.0.1) which depends on
          `numpy`

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it wasnt flakiness after all. Yeah this workaround sounds OK for now!

@potiuk
Copy link
Member Author

potiuk commented Jan 30, 2025

So it wasnt flakiness after all. Yeah this workaround sounds OK for now!

It's a good long term one actually- because we prevent installing old versions that are not installable on newer python versions :)

@potiuk potiuk merged commit 63d3602 into apache:main Jan 30, 2025
21 checks passed
@potiuk potiuk deleted the limit-numpy-for-beam branch January 30, 2025 12:59
@potiuk
Copy link
Member Author

potiuk commented Jan 30, 2025

Merging to get next canary builds to run it

ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Jan 30, 2025
Separating beam out reveald that the lower limit for numpy migh
lead to installation problems on newer package managers and python
versions (lack of distutils installed by default in Python 3.12 in
causes older versions of numpy to not build and uv / pip trying to
install it will fail.

This PR lower-binds numpy to mid-2023 version.
@jscheffl
Copy link
Contributor

Cool! Yeah, did not notice this but also in example DAGs numpy < 1.26 failed with Python 3.12. Good that this is also fixed here!

niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
Separating beam out reveald that the lower limit for numpy migh
lead to installation problems on newer package managers and python
versions (lack of distutils installed by default in Python 3.12 in
causes older versions of numpy to not build and uv / pip trying to
install it will fail.

This PR lower-binds numpy to mid-2023 version.
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
Separating beam out reveald that the lower limit for numpy migh
lead to installation problems on newer package managers and python
versions (lack of distutils installed by default in Python 3.12 in
causes older versions of numpy to not build and uv / pip trying to
install it will fail.

This PR lower-binds numpy to mid-2023 version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants