Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set minimum dependencies for apache-beam deps on Py 3.12+3.13 #46321

Merged
merged 1 commit into from
Jan 31, 2025

Conversation

ashb
Copy link
Member

@ashb ashb commented Jan 31, 2025

Set minimum dependencies for apache-beam on Py 3.12+3.13

Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Copy link
Member

potiuk commented Jan 31, 2025

Cool !

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it will work, aligning with the last fix for beam

@ashb
Copy link
Member Author

ashb commented Jan 31, 2025

Odd, that didn't seem to work:

  + uv pip install --python /usr/local/bin/python --resolution lowest-direct --upgrade --editable '.[apache-beam]' --editable ./task_sdk
  Using Python 3.12.8 environment at: /usr/local
  Resolved 179 packages in 4.73s
    × Failed to download and build `apache-beam==2.55.0`
    ├─▶ Failed to install requirements from `build-system.requires`
    ├─▶ Failed to build `grpcio-tools==1.53.0`

@ashb
Copy link
Member Author

ashb commented Jan 31, 2025

I'm going to set the min version of apache-beam itself instead.

@potiuk
Copy link
Member

potiuk commented Jan 31, 2025

Ah. This is because apache-beam 2.55.0 has grpcio.tools as "build requirements" not "requirements". In a modern build packaging - when a frontend is building package (because it does not have corresponding binary wheel) it will perform the build of that package from .sdist in "isolated build environment" -> where it downloads and sets up temporary venv with the dependencies specified in the "build requirements" of that package.

From the error looks like apache-beam 2.55.0 has grpcio-tools 1.53.0 set as their build requirements.

@potiuk
Copy link
Member

potiuk commented Jan 31, 2025

So yeah - we should limit apache.beam not grpc-tools in this case.

@ashb ashb force-pushed the possible-beam-downgrade-fix branch from b80fcac to 488e9db Compare January 31, 2025 16:46
Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.
@ashb ashb force-pushed the possible-beam-downgrade-fix branch from 488e9db to 91ef55a Compare January 31, 2025 17:00
@ashb ashb merged commit 41aae33 into main Jan 31, 2025
156 checks passed
@ashb ashb deleted the possible-beam-downgrade-fix branch January 31, 2025 18:03
utkarsharma2 pushed a commit that referenced this pull request Jan 31, 2025
Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.

This updates the min version to the min supported version with wheels for 3.12
Prab-27 pushed a commit to Prab-27/airflow that referenced this pull request Jan 31, 2025
Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.

This updates the min version to the min supported version with wheels for 3.12
amoghrajesh pushed a commit to astronomer/airflow that referenced this pull request Feb 3, 2025
Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.

This updates the min version to the min supported version with wheels for 3.12
dabla pushed a commit to dabla/airflow that referenced this pull request Feb 3, 2025
Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.

This updates the min version to the min supported version with wheels for 3.12
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.

This updates the min version to the min supported version with wheels for 3.12
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
Older versions of apache-beam package depend upon gprcio-tools, but the
version requirement they specify doesn't work on Py 3.12, but the metadata
doesn't say that, so it tries to compile it and fails.

This updates the min version to the min supported version with wheels for 3.12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants