Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(restore-indices): add endpoint for restore indices, add basic check for graph #5805

Conversation

anshbansal
Copy link
Collaborator

@anshbansal anshbansal commented Sep 1, 2022

  • Moved bulk of code from restore indices into Entity Service
  • Added endpoint to be able to run restore indices via API
  • Add a simple check for graph consistency of dataPlatformInstance

image

New command line in case there is a diff between mysql and elastic

datahub check graph-consistency
[WARN dataPlatformInstance] Expected to have 4053 but found 4029. Missing % = 0.59

New command line in case there is no diff between mysql and elastic

datahub check graph-consistency
[SUCCESS dataPlatformInstance] Expected and actual are 4054

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions
Copy link

github-actions bot commented Sep 1, 2022

Unit Test Results (build & test)

517 tests  ±0   517 ✔️ ±0   9m 21s ⏱️ +2s
121 suites ±0       0 💤 ±0 
121 files   ±0       0 ±0 

Results for commit 418810d. ± Comparison against base commit c91e29b.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Sep 1, 2022

Unit Test Results (metadata ingestion)

       8 files  ±0         8 suites  ±0   55m 18s ⏱️ -43s
   671 tests ±0     666 ✔️  - 2  3 💤 ±0  0 ±0  2 🔥 +2 
1 342 runs  ±0  1 334 ✔️  - 2  6 💤 ±0  0 ±0  2 🔥 +2 

For more details on these errors, see this check.

Results for commit 418810d. ± Comparison against base commit c91e29b.

♻️ This comment has been updated with latest results.

@anshbansal anshbansal added the devops PR or Issue related to DataHub backend & deployment label Sep 1, 2022
rowsMigrated++;
}

try {
Copy link
Collaborator

@RyanHolstien RyanHolstien Sep 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove sleep, this is unneeded. It was in historically to "guarantee a write went through", but it doesn't really make sense to do so.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do

public long sendMessageMs = 0;

@Override
public String toString() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can also just be done with lombok

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do

}
}

public RestoreIndicesArgs setAspectName(String aspectName) {
Copy link
Collaborator

@RyanHolstien RyanHolstien Sep 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can just use @Data from lombok

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That will cause return type to be void. I wanted a builder pattern for ease of use.

@anshbansal anshbansal merged commit b643b34 into datahub-project:master Sep 2, 2022
@anshbansal anshbansal deleted the restore-indices-endpoint-graph-consistency branch September 2, 2022 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
devops PR or Issue related to DataHub backend & deployment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants