OpenLineage Spark extension interfaces without runtime dependency hell #2809

ddebowczyk92 · 2024-06-28T15:06:43Z

Problem

Closes: #2784

Your change modifies the core OpenLineage model
Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

New Spark extension interfaces without runtime dependency hell

Checklist

You've signed-off your work
Your pull request title follows our guidelines
Your changes are accompanied by tests (if relevant)
Your change contains a small diff and is self-contained
You've updated any relevant documentation (if relevant)
Your comment includes a one-liner for the changelog about the specific purpose of the change (not required for changes to tests, docs, or CI config)
You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

pawel-big-lebowski

Massive amount of work on a highly desired feature.
I'm glad to see this being done with a dynamite test to verify integration is working.
Left few comments / questions to let me understand better the change.

Phaty PR. Thank you @ddebowczyk92 for bringing this.

.circleci/workflows/openlineage-spark.yml

pawel-big-lebowski · 2024-07-01T11:50:47Z

...o/openlineage/shaded/spark/extension/v1/lifecycle/plan/SparkOpenLineageExtensionVisitor.java

+/* Copyright 2018-2024 contributors to the OpenLineage project
+/* SPDX-License-Identifier: Apache-2.0
+*/
+package io.openlineage.shaded.spark.extension.v1.lifecycle.plan;


any reason for having io.openlineage.shaded.spark.extension.v1 instead of io.openlineage.spark.shaded.extension.v1 ?

Maybe even different package name? io.openlineage.spark suggests it's a part of openlineage-spark mvn package.

io.openlineage.interfaces.spark?

.circleci/config.yml

integration/spark-interfaces-shaded/README.md

pawel-big-lebowski · 2024-07-02T07:24:30Z

integration/spark-interfaces-unshaded/gradle.properties

@@ -0,0 +1 @@
+version=1.0.0-SNAPSHOT


is this deliberate? how are you going to release it?

I think unshaded should have static version, manually bumped if needed. Release should skip it if the particular version is released.

ok. can we then release it manually and merge to main version version=1.0.0? I don't mind the approach taken but I think snapshot version should be avoided.

.../main/java/io/openlineage/spark/agent/lifecycle/SparkOpenLineageExtensionVisitorWrapper.java

.circleci/continue_config.yml

...o/openlineage/shaded/spark/extension/v1/lifecycle/plan/SparkOpenLineageExtensionVisitor.java

.../main/java/io/openlineage/spark/agent/lifecycle/SparkOpenLineageExtensionVisitorWrapper.java

integration/spark-interfaces-shaded/README.md

integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageContext.java

Signed-off-by: Dominik Dębowczyk <[email protected]>

codecov-commenter · 2024-07-04T15:41:50Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.52%. Comparing base (a31cc8d) to head (d1d1a13).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2809   +/-   ##
=======================================
  Coverage   85.52%   85.52%           
=======================================
  Files          58       58           
  Lines        3413     3413           
=======================================
  Hits         2919     2919           
  Misses        494      494

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Dominik Dębowczyk <[email protected]>

pawel-big-lebowski

Looks good to me.

pawel-big-lebowski · 2024-07-05T12:21:44Z

.circleci/continue_config.yml

+              ignore: /.*/
+          context: release
+          requires:
+            - release-client-java


is this required?

mobuchowski

Thanks @ddebowczyk92 for such a detailed work 👍

mobuchowski · 2024-07-05T12:41:39Z

integration/spark-extension-entrypoint/README.md

@@ -0,0 +1,38 @@
+# Spark Extension Entrypoint


This needs page in https://openlineage.io/docs/ (outside of this PR)

mobuchowski · 2024-07-05T12:58:40Z

integration/spark/shared/build.gradle

@@ -44,6 +43,7 @@ dependencies {
    api("io.openlineage:openlineage-java:${project.version}")
    api("io.openlineage:openlineage-sql-java:${project.version}")
    api("io.micrometer:micrometer-core:${micrometerVersion}")
+    api("io.openlineage:spark-extension-entrypoint:1.0.0-SNAPSHOT")


chicken and egg release problem, right?

I think we just need to release entrypoint separately first, before next regular OL release.

ddebowczyk92 requested a review from a team as a code owner June 28, 2024 15:06

boring-cyborg bot added area:ci CI area:documentation Improvements or additions to documentation area:tests Testing code language:java Uses Java programming language labels Jun 28, 2024

ddebowczyk92 force-pushed the add-spark-interfaces branch 6 times, most recently from 9738a63 to 7c1bb82 Compare July 1, 2024 14:08

pawel-big-lebowski reviewed Jul 2, 2024

View reviewed changes

mobuchowski reviewed Jul 3, 2024

View reviewed changes

integration/spark-interfaces-shaded/README.md Outdated Show resolved Hide resolved

mobuchowski reviewed Jul 3, 2024

View reviewed changes

integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageContext.java Outdated Show resolved Hide resolved

ddebowczyk92 force-pushed the add-spark-interfaces branch 16 times, most recently from 6682bb3 to cd3bd95 Compare July 4, 2024 11:57

ddebowczyk92 added 9 commits July 4, 2024 16:19

[SPARK] Create spark-interfaces-unshaded module

a6eb68f

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Create spark-interfaces-shaded module

ba08859

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Add spark-interfaces-shaded support

bb2e795

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Fix licenses

e5fa51a

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Fix pmd errors

ad6b196

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Apply spotless

caf19d5

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Rename modules and packages

4a20938

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Fix tests

e20a582

Signed-off-by: Dominik Dębowczyk <[email protected]>

[SPARK] Add versions support

d1d1a13

Signed-off-by: Dominik Dębowczyk <[email protected]>

ddebowczyk92 force-pushed the add-spark-interfaces branch from cd3bd95 to d1d1a13 Compare July 4, 2024 14:19

[SPARK] Rename spark-extension-frontend to spark-extension-entrypoint

84f1a88

Signed-off-by: Dominik Dębowczyk <[email protected]>

pawel-big-lebowski approved these changes Jul 5, 2024

View reviewed changes

.circleci/continue_config.yml

ignore: /.*/

context: release

requires:

- release-client-java

Copy link

Collaborator

pawel-big-lebowski Jul 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this required?

mobuchowski approved these changes Jul 5, 2024

View reviewed changes

mobuchowski merged commit b736dab into OpenLineage:main Jul 5, 2024
70 checks passed

dolfinus mentioned this pull request Jul 9, 2024

[SPARK] Spark extension Java interfaces #2675

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenLineage Spark extension interfaces without runtime dependency hell #2809

OpenLineage Spark extension interfaces without runtime dependency hell #2809

ddebowczyk92 commented Jun 28, 2024 •

edited

Loading

pawel-big-lebowski left a comment

pawel-big-lebowski Jul 1, 2024

mobuchowski Jul 3, 2024

pawel-big-lebowski Jul 2, 2024

mobuchowski Jul 3, 2024

pawel-big-lebowski Jul 4, 2024

codecov-commenter commented Jul 4, 2024

pawel-big-lebowski left a comment

pawel-big-lebowski Jul 5, 2024

mobuchowski left a comment

mobuchowski Jul 5, 2024

mobuchowski Jul 5, 2024

		@@ -0,0 +1 @@
		version=1.0.0-SNAPSHOT

OpenLineage Spark extension interfaces without runtime dependency hell #2809

OpenLineage Spark extension interfaces without runtime dependency hell #2809

Conversation

ddebowczyk92 commented Jun 28, 2024 • edited Loading

Problem

One-line summary:

Checklist

pawel-big-lebowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 4, 2024

Codecov Report

pawel-big-lebowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mobuchowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddebowczyk92 commented Jun 28, 2024 •

edited

Loading