Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Great Expectations v3 integration with airflow operator #689

Merged
merged 6 commits into from
Apr 21, 2022

Conversation

collado-mike
Copy link
Contributor

Problem

main branch builds are failing due to integration test failures with the update to support Great Expectation's v3 API - see https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/2981/workflows/15bd0993-0423-4958-927d-a5d427993894/jobs/27943

Updated unit tests to catch the issue so that we can catch these in fork builds as well as main ones.

Note: All schema changes require discussion. Please link the issue for context.

  • Your change modifies the core OpenLineage model
  • Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)

@collado-mike collado-mike requested a review from wslulciuc April 19, 2022 22:21
@collado-mike collado-mike force-pushed the airflow_great_expectations_fix branch from 6a51634 to 3838142 Compare April 20, 2022 04:34
@mobuchowski
Copy link
Member

@collado-mike I had earlier PR that also fixes the same thing. #688

Can we merge the test part of this PR?

@collado-mike collado-mike force-pushed the airflow_great_expectations_fix branch from 3838142 to af5c068 Compare April 20, 2022 17:52
@collado-mike
Copy link
Contributor Author

Can we merge the test part of this PR?

Done. I kept the simplification in the _get_sql_table method, as I didn't see any point in having the same logic twice.

@collado-mike collado-mike enabled auto-merge (squash) April 20, 2022 19:09
@collado-mike collado-mike disabled auto-merge April 20, 2022 19:09
@mobuchowski mobuchowski merged commit 68d77df into main Apr 21, 2022
@mobuchowski mobuchowski deleted the airflow_great_expectations_fix branch April 21, 2022 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants