[SPARK] fix removePathPattern behaviour

[pawel-big-lebowski]

Problem

removepath pattern feature is not applied all the time. The method is called when constructing DatasetIdentifier through PathUtils which is not the case all the time.

Closes: #2335

Solution

• Move removePattern to another place in codebase which is always run,
• Write integration test for this.

Note: All schema changes require discussion. Please link the issue for context.

• Your change modifies the core OpenLineage model
• Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

Checklist

• You've signed-off your work
• Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
• You've added a header to source files (if relevant)

---

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

[mobuchowski]

One comment but lgtm.

[mobuchowski]

Duplicate?

[pawel-big-lebowski]

this got moved to java client some time ago

[mobuchowski]

Maybe List<? extends Dataset> and implement this generically with passed dataset builder? Nit, do what you think is best.

[pawel-big-lebowski]

I have tried this approach but did not succeed.
Creating OutputDataset and InputDataset differs significantly and these have to be two separate methods.
The iterating part, that goes through the list, is common.
So a common generic method would need to call methods per OutputDataset or InputDataset, which didn't look as an improvement to me.