Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added support for iceberg rest catalog #1963

Conversation

juancappi
Copy link
Contributor

Problem

Currently, Spark Iceberg integration for Openlineage only supports hive and hadoop catalogs. Iceberg also has RestCatalog. It would be good to have it supported for Openlineage also as it is gaining popularity.

Implements: #1940

Solution

Add rest to the existing options of hive and hadoop in IcebergHandler.getDatasetIdentifier()
It's a backwards-compatible change.

  • Your change modifies the core OpenLineage model
  • Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary: add support for iceberg rest catalog in IcebergHandler

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

@boring-cyborg
Copy link

boring-cyborg bot commented Jul 7, 2023

Thanks for opening your first OpenLineage pull request! We appreciate your contribution. If you haven't already, please make sure you've reviewed our guide for new contributors (https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md).

@pawel-big-lebowski
Copy link
Collaborator

Hi @juancappi,

Thank you for submitting your PR. Circle-ci is failing for pre-commit test which was fixed last week. Could you please rebase your PR onto current main?

Juan Cappi added 2 commits July 10, 2023 11:31
@juancappi juancappi force-pushed the feature/1940_support_rest_catalog_iceberg_handler branch from 3c53a85 to a61d174 Compare July 10, 2023 14:36
@juancappi
Copy link
Contributor Author

@pawel-big-lebowski rebase done. Circle-ci still seems to be failing in the same place.

@pawel-big-lebowski pawel-big-lebowski merged commit bc1f525 into OpenLineage:main Jul 11, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Jul 11, 2023

Great job! Congrats on your first merged pull request in OpenLineage!

@pawel-big-lebowski
Copy link
Collaborator

Congrats @juancappi 🥇 🚀 👍 💯

@juancappi
Copy link
Contributor Author

Thank you @pawel-big-lebowski !

@juancappi juancappi deleted the feature/1940_support_rest_catalog_iceberg_handler branch July 11, 2023 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants