feat(lineage) Add column-level impact analysis feature #6272
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aditya-radhakrishnan
approved these changes
Oct 25, 2022
| Urn resourceUrn = Urn.createFromString(schemaFieldUrn.getEntityKey().get(0)); | ||
| result.setParent(UrnToEntityMapper.map(resourceUrn)); | ||
| } catch (Exception e) { | ||
| log.error("Error converting schemaField parent urn string to Urn", e); |
There was a problem hiding this comment.
There's no issue if we can't convert right? It's still safe to return the mapped object without this field?
There was a problem hiding this comment.
ah that's a good catch - SchemaFieldEntity is expecting parent to not be null here. so I think our options are to either make parent nullable or raise an error if this situation occurs (which it shouldn't). I don't think it really makes sense to have parent be null since the schemaField should always have a valid parent reference urn in its urn.. so i'm leaning towards raising an error. what do you think?
There was a problem hiding this comment.
talked IRL and agreed on throwing an exception here
| @@ -2343,21 +2343,31 @@ type KeyValueSchema { | |||
| Standalone schema field entity. Differs from the SchemaField struct because it is not directly nested inside a | |||
| schema field | |||
| """ | |||
| type SchemaFieldEntity { | |||
| type SchemaFieldEntity implements Entity { | |||
| @@ -55,4 +55,29 @@ describe("impact analysis", () => { | |||
| cy.contains("User Creations").should("not.exist"); | |||
| cy.contains("User Deletions"); | |||
| }); | |||
|
|
|||
| it("can view column level impact analysis and turn it off", () => { | |||
There was a problem hiding this comment.
Amazing, thank you!
added 2 commits
October 25, 2022 17:38
cccs-tom
pushed a commit
to CybercentreCanada/datahub
that referenced
this pull request
Nov 18, 2022
cccs-tom
pushed a commit
to CybercentreCanada/datahub
that referenced
this pull request
Nov 18, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the ability to view impact analysis specifically at the column level. You can now select a column to view impact analysis on from the schema tab with a new menu or select from a dropdown on the lineage tab. You'll see the impacted columns and you can even click on the column name in order to see the whole path for how you get between the two columns.
This involved some changes to the how we index upstream lineage. We now check if the upstream lineage aspect that we're indexing has fineGrainedLineage, and if so, we draw edges manually between the two
schemaFields. That way we can query to get edges based on aschemaFieldurn.Unforunately this means that for those users that already have fineGrainedLineage ingested, they will need to restore their indices in order to get old fineGrainedLineage data to work with impact analysis
Here's some screenshots for how this looks in action:
The new column menu
Viewing impact analysis for the
emailcolumnThe paths modal to see the full path between entities and their columns (all of these happen to be named
email)Checklist