Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RowStats for multiphase parallel indexing task #12280

Merged
merged 14 commits into from
Mar 2, 2022
Merged

RowStats for multiphase parallel indexing task #12280

merged 14 commits into from
Mar 2, 2022

Conversation

tejaswini-imply
Copy link
Member

@tejaswini-imply tejaswini-imply commented Feb 24, 2022 •
edited
Loading

Displays RowStats in reports for multi-phase parallel indexing task.

Added support for RowStats in druid/indexer/v1/task/{task_id}/reports api for multi-phase parallel indexing task.

Key changed/added classes in this PR
  • GeneratedPartitionsReport.java
  • GeneratedPartitionsMetadataReport.java
  • ParallelIndexSupervisorTask.java
  • PartialHashSegmentGenerateTask.java
  • PartialRangeSegmentGenerateTask.java
  • PartialSegmentGenerateTask.java
  • ParallelIndexSupervisorTaskTest.java

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@tejaswini-imply tejaswini-imply changed the title RowStats for multiphase parallel task RowStats for multiphase parallel indexing task Feb 24, 2022
@kfaraz kfaraz marked this pull request as ready for review March 1, 2022 06:02
kfaraz
kfaraz reviewed Mar 1, 2022
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested minor changes. Overall LGTM.

...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
kfaraz
kfaraz reviewed Mar 1, 2022
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes, @tejaswini-imply . I have added a couple more suggestions.

...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java Outdated Show resolved Hide resolved
kfaraz
kfaraz approved these changes Mar 1, 2022
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@kfaraz
Copy link
Contributor

kfaraz commented Mar 2, 2022

Merging as CI failure is unrelated.

@kfaraz kfaraz merged commit 1af4c9c into apache:master Mar 2, 2022
@kfaraz kfaraz mentioned this pull request Mar 7, 2022
Merged
9 tasks
@abhishekagarwal87 abhishekagarwal87 added this to the 0.23.0 milestone May 11, 2022
@abhishekagarwal87 abhishekagarwal87 mentioned this pull request May 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kfaraz kfaraz kfaraz approved these changes

Assignees
No one assigned
Labels
Area - Batch Ingestion Improvement
Projects
None yet
Milestone
0.23.0
Development

Successfully merging this pull request may close these issues.
4 participants
@tejaswini-imply @kfaraz @abhishekagarwal87 @clintropolis