fix(ui) Handle encoded schemaField urns on the frontend #6321
Conversation
github-actions
bot
added
ingestion
| @@ -116,9 +116,17 @@ export function filterColumns( | |||
| } | |||
| } | |||
|
|
|||
| export function decodeSchemaField(fieldPath: string) { | |||
| @@ -25,7 +26,7 @@ export default function DisplayedColumns({ displayedColumns }: Props) { | |||
| return ( | |||
| <ColumnNameWrapper> | |||
| {entity.type === EntityType.SchemaField | |||
| ? downgradeV2FieldPath((entity as SchemaFieldEntity).fieldPath) | |||
| ? decodeSchemaField(downgradeV2FieldPath((entity as SchemaFieldEntity).fieldPath) || '') | |||
There was a problem hiding this comment.
Should we decode after downgrading or before?
There was a problem hiding this comment.
I don't think it would really matter as decoding it just converts a few encoded characters back to normal ((, ), and ,) and none of those should be affected by the V2 field path. I believe decoding after downgrading would be safest so then we're decoding what we want to show in the UI.
| @@ -1,6 +1,8 @@ | |||
| import urllib.parse | |||
| from typing import List | |||
|
|
|||
| # NOTE: Frontend relies on encoding these three characters. Specifically, we decode and encode schema fields for column level lineage. | |||
There was a problem hiding this comment.
Question - In such cases, how will these column fields appear in
a) Schema Tab
b) Stats Tab
Do we also need that encoding there? If yes, maybe consider adding in this PR, or marking TODOs.
There was a problem hiding this comment.
great question - I believe these tabs do not need to be decoded, since the tables in them are reading from fields in mysql that have the fieldPath stored in them, not from schemaField urns that we generate and encode.
So the only time we have encoded fieldPaths is when we're actually grabbing the fieldPath from the schema field urn - and we only have those stored in upstreamLineage, inputFields, and now that you're calling this out, I discovered we're also storing them in DatasetAssertionInfo and under foreignKeys in SchemaMetadata - going to add some coverage for that just in case
Our ingestion crew had to encode
schemaFieldurns so that thefieldPathof the schema is encoding a few special characters ((,), and,). We need to decode those on the frontend wherever we display fieldPaths that we get fromupstreamLineageaspects. We also need to encode fieldPaths for CLL impact analysis since the urn in our graph index will have the encoded urns.I also added a comment where we ingestion calls this encoding to make sure they are always in sync with the frontend code as well.
Checklist