Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Dataproc operators fail to import without OpenLineage #46561

Merged
merged 1 commit into from
Feb 7, 2025

Conversation

kacpermuda
Copy link
Contributor

related: #46478

This PR does three things:

  1. Moves top level import from Google's OpenLineage utils into operator's execute method, to avoid issues with ImportError, should they happen in the future.
  2. Adds try/except for OpenLineage code in DataProc operators, so that any error related to OpenLineage does not affect execution of DataProc.
  3. Moves BigQueryJobRunFacet to a separate file and add try/except on class definition, in case OpenLineage is not available (similar logic as in common.compat provider). I've also moved the import of this facet into specific method in _OpenLineageMixin, so it's no longer top level import.

I've tested it manually on the environment without OpenLineage and the error described in the issue is gone after applying changes included in this PR.

CC @mobuchowski


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added the provider:google Google (including GCP) related issues label Feb 7, 2025
@potiuk
Copy link
Member

potiuk commented Feb 7, 2025

Nice. Some mypy checks but it looks good :)

@potiuk
Copy link
Member

potiuk commented Feb 7, 2025

Quick fix @kacpermuda :)

@kacpermuda kacpermuda force-pushed the fix-google-ol-import-error branch from 8d9f817 to 682ff6c Compare February 7, 2025 15:43
@kacpermuda
Copy link
Contributor Author

@potiuk let's hold on with merging until we get @mobuchowski approval. We discussed few ways to solve this, and I want to hear his opinion.

@potiuk
Copy link
Member

potiuk commented Feb 7, 2025

Sure

@mobuchowski
Copy link
Contributor

No need to be a single point of failure 🙂
PR looks good to me.

@mobuchowski mobuchowski merged commit 4f2743c into apache:main Feb 7, 2025
61 checks passed
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
@kacpermuda kacpermuda deleted the fix-google-ol-import-error branch February 10, 2025 06:56
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants