Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Log histograms should be faster #148682

Open
StephanErb opened this issue Jan 10, 2023 · 2 comments
Open

Log histograms should be faster #148682

StephanErb opened this issue Jan 10, 2023 · 2 comments
Labels
Feature:Logs UI Logs UI feature Team:obs-ux-logs Observability Logs User Experience Team Team:Observability Team label for Observability Team (for things that are handled across all of observability)

Comments

@StephanErb
Copy link

StephanErb commented Jan 10, 2023 •
edited
Loading

Kibana is showing histogram of log messages over time in three different places:

  • in Discover
  • in the sidebar/minimap of the logs streaming ui
  • on the Observability home page as log rate per minute

The Discover rendering is notably faster than the other two options. On our clusters rendering an logs histogram for a 24h time frame takes about 5-8s in Discover, but 20-40s in the other two. A 24h timeframe includes 500M-1000M documents.

I believe the faster Discover rendering is due to elastic/elasticsearch#81322. The other histogram calls don't seem to be profiting from the same optimization yet, as far as I understand. Tweaking the query might help here.

In addition, this could probably be speed up further with random sampling. Histograms only show a high-level overview over time. Accuracy is less important than getting a result quickly.

Also relates to #148670
@botelastic botelastic bot added the needs-team Issues missing a team label label Jan 10, 2023
@StephanErb StephanErb changed the title Log histograms should use date_histogram and random sampling Log histograms should be faster Jan 11, 2023
@dej611 dej611 added Feature:Logs UI Logs UI feature Team:Observability Team label for Observability Team (for things that are handled across all of observability) labels Jan 11, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/unified-observability (Team:Observability)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Jan 11, 2023
@gbamparop gbamparop added the Team:obs-ux-logs Observability Logs User Experience Team label Nov 9, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature:Logs UI Logs UI feature Team:obs-ux-logs Observability Logs User Experience Team Team:Observability Team label for Observability Team (for things that are handled across all of observability)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@StephanErb @dej611 @gbamparop @elasticmachine