-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Metric to report time spent fetching and analyzing segments #14752
Conversation
@@ -904,6 +928,8 @@ static class ExistingSegmentAnalyzer | |||
// For processMetricsSpec: | |||
private final Set<List<AggregatorFactory>> aggregatorFactoryLists = new HashSet<>(); | |||
|
|||
private long processingTimeMillis = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed. I had thought perhaps a metric for all the time spent in this class would be interesting, but decided against implementing it as it would have resulted in a few new metrics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed metrics.md and left some suggestions.
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: Katya Macedo <[email protected]>
Description
Add a new metric that is emitted by the compaction task that shows how much time it spent fetching and analyzing segments. When a datasource is very fragmented, this could be a large amount of time. This metric will be helpful to operators as they can monitor this to see if they would benefit from explicitly specifying the schema in the auto compaction / manual compaction config if their use case allows it.
Here is a sample of what the metric would look like
Release note
NEW:
compact/segmentAnalyzer/fetchAndProcessMillis
is now reported by compaction tasks to indicate how much time was spent fetching and processing segments to infer the schemaThis PR has: