Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingestion) Add more info to glue entities #5874

Merged
merged 11 commits into from
Sep 14, 2022

Conversation

skrydal
Copy link
Collaborator

@skrydal skrydal commented Sep 8, 2022

This PR adds subTypes=["Table"] to glue datasets (tables) and also ingest Description of glue database (similarly to Description of the tables).

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions
Copy link

github-actions bot commented Sep 8, 2022

Unit Test Results (metadata ingestion)

       8 files  ±0         8 suites  ±0   56m 36s ⏱️ + 1m 45s
   705 tests  - 4     702 ✔️  - 4  3 💤 ±0  0 ±0 
1 410 runs   - 8  1 404 ✔️  - 8  6 💤 ±0  0 ±0 

Results for commit ccf0ae4. ± Comparison against base commit 15a33fa.

This pull request removes 5 and adds 1 tests. Note that renamed tests count towards both.
tests.unit.stateful_ingestion.provider.test_datahub_ingestion_reporting_provider.TestDatahubIngestionReportingProvider ‑ test_provider
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[reporting_bad_config]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[reporting_default]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[reporting_valid_full_config]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[reporting_valid_simple_config]
tests.unit.config.test_config_enum ‑ test_config_enum

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Sep 8, 2022

Unit Test Results (build & test)

571 tests  ±0   571 ✔️ ±0   14m 24s ⏱️ -28s
141 suites ±0       0 💤 ±0 
141 files   ±0       0 ±0 

Results for commit ccf0ae4. ± Comparison against base commit 15a33fa.

♻️ This comment has been updated with latest results.

@skrydal skrydal force-pushed the add-more-info-to-glue-entities branch from a1783dc to 3d3c8ac Compare September 8, 2022 19:18
@shirshanka shirshanka added community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata labels Sep 8, 2022
@skrydal skrydal force-pushed the add-more-info-to-glue-entities branch from 3d3c8ac to 3bd8ceb Compare September 9, 2022 07:58
@skrydal skrydal force-pushed the add-more-info-to-glue-entities branch from 7969081 to c1bd52e Compare September 11, 2022 10:35
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a minor tweak around the mcp generation logic. Overall LGTM.

One thing before we can merge this: I need to check if the subType piece is case-sensitive - we have used table elsewhere in the codebase but Table here, so I want to verify that it won't cause any issues.

@skrydal
Copy link
Collaborator Author

skrydal commented Sep 14, 2022

@hsheth2 thank you for the comment, I am afraid introduced changes are breaking the tests, there are fails which weren't there previously: https://github.com/datahub-project/datahub/actions/runs/3049282407/jobs/4915193424 - I would fix it but I am not sure I understand why they were introduced.

@hsheth2
Copy link
Collaborator

hsheth2 commented Sep 14, 2022

That's my bad - fixed up the tests

@hsheth2 hsheth2 merged commit f61a040 into datahub-project:master Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants