feat(ingest): add column-level lineage support for snowflake

[mayurinehate]

Other changes

• fix missing view upstream lineage
• remove customProperties with column-lineage info

Note:

• Direct Column level lineage is available only for tables(as downstream) and not for views(as downstream).
• The bag of columns lineage as available in customProperties earlier is not used as fine grained lineage, but could be used if required with small code change, if required.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[github-actions]

Unit Test Results (metadata ingestion)

       8 files  ±0         8 suites  ±0   1h 0m 58s ⏱️ -33s
   719 tests +2     716 ✔️ +2  3 💤 ±0  0 ❌ ±0 
1 440 runs  +4  1 434 ✔️ +4  6 💤 ±0  0 ❌ ±0 

Results for commit 75e71c7. ± Comparison against base commit 0728656.

This pull request removes 8 and adds 10 tests. Note that renamed tests count towards both.

tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/bad_complex_variable_expansion.yml-None-env6-MissingClosingBrace]
tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/bad_extension.whatevenisthis-None-env5-ConfigurationError]
tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/bad_variable_expansion.yml-None-env3-ParameterNullOrNotSet]
tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/basic.toml-golden_config1-env1-None]
tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/basic.yml-golden_config0-env0-None]
tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/complex_variable_expansion.yml-golden_config7-env7-None]
tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/simple_variable_expansion.yml-golden_config2-env2-None]
tests.unit.config.test_config_loader ‑ test_load[tests/unit/config/this_file_does_not_exist.yml-None-env4-ConfigurationError]

tests.integration.tableau.test_tableau_ingest ‑ test_tableau_usage_stat
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/bad_complex_variable_expansion.yml-env3-MissingClosingBrace]
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/bad_extension.whatevenisthis-env2-ConfigurationError]
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/bad_variable_expansion.yml-env0-ParameterNullOrNotSet]
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/this_file_does_not_exist.yml-env1-ConfigurationError]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/basic.toml-golden_config1-env1-None]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/basic.yml-golden_config0-env0-referenced_env_vars0]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/complex_variable_expansion.yml-golden_config3-env3-referenced_env_vars3]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/simple_variable_expansion.yml-golden_config2-env2-referenced_env_vars2]
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_supported_encodings

♻️ This comment has been updated with latest results.

[github-actions]

Unit Test Results (build & test)

584 tests  +16   580 ✔️ +16   12m 54s ⏱️ -2s
143 suites +  3       4 💤 ±  0 
143 files   +  3       0 ❌ ±  0 

Results for commit 75e71c7. ± Comparison against base commit 0728656.

This pull request removes 9 and adds 25 tests. Note that renamed tests count towards both.

com.datahub.authentication.user.NativeUserServiceTest ‑ testConstructor
com.datahub.authentication.user.NativeUserServiceTest ‑ testCreateNativeUserInviteTokenDoesNotExist
com.datahub.authentication.user.NativeUserServiceTest ‑ testGenerateNativeUserInviteTokenPasses
com.datahub.authentication.user.NativeUserServiceTest ‑ testGetNativeUserInviteTokenInviteTokenDoesNotExistPasses
com.datahub.authentication.user.NativeUserServiceTest ‑ testGetNativeUserInviteTokenPasses
com.linkedin.datahub.graphql.resolvers.user.CreateNativeUserInviteTokenResolverTest ‑ testFailsCannotManageUserCredentials
com.linkedin.datahub.graphql.resolvers.user.CreateNativeUserInviteTokenResolverTest ‑ testPasses
com.linkedin.datahub.graphql.resolvers.user.GetNativeUserInviteTokenResolverTest ‑ testFailsCannotManageUserCredentials
com.linkedin.datahub.graphql.resolvers.user.GetNativeUserInviteTokenResolverTest ‑ testPasses

com.datahub.authentication.invite.InviteTokenServiceTest ‑ getInviteToken
com.datahub.authentication.invite.InviteTokenServiceTest ‑ getInviteTokenEmptySearchResult
com.datahub.authentication.invite.InviteTokenServiceTest ‑ getInviteTokenNoInviteTokenAspect
com.datahub.authentication.invite.InviteTokenServiceTest ‑ getInviteTokenNullEntity
com.datahub.authentication.invite.InviteTokenServiceTest ‑ getInviteTokenRegenerate
com.datahub.authentication.invite.InviteTokenServiceTest ‑ getInviteTokenRoleUrnDoesNotExist
com.datahub.authentication.invite.InviteTokenServiceTest ‑ testGetInviteTokenRole
com.datahub.authentication.invite.InviteTokenServiceTest ‑ testGetInviteTokenRoleEmptyAspectMap
com.datahub.authentication.invite.InviteTokenServiceTest ‑ testGetInviteTokenRoleNoRole
com.datahub.authentication.invite.InviteTokenServiceTest ‑ testGetInviteTokenRoleNullEntity
…

♻️ This comment has been updated with latest results.

[shirshanka]

what happens if this bombs?

[mayurinehate]

hmm, I don't expect bombing since the VIEW_COLUMNS column is equivalent to columns json array as per snowflake access history view documentation and the dict to class conversion is protected already.

[hsheth2]

have some questions around __eq__ and __hash__

[hsheth2]

LGTM