Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Stack Monitoring] Add log-* index pattern support on SM UI #139121

Merged

Conversation

crespocarlos
Copy link
Contributor

@crespocarlos crespocarlos commented Aug 18, 2022

Summary

This PR adds logs-* index pattern support on places in the SM UI where it queries filebeat-* modules, in order to also work with integration packages

closes #120416

Screenshots

Cluster Overview UI
image

Elasticsearch overview
image

Indices view
image

Testing

Make sure you have elastic-package installed.

  • This change works with a corresponding PR in the Integrations package. Make sure to follow the instructions in ES package log pipelines integrations#4033 to setup the required services
  • Configure your local kibana to point to elastic-package stack. See howto
  • Connect to local Kibana at http://localhost:5602 and navigate to the Stack Monitoring UI. Logs should show up in the Elasticsearch views

@crespocarlos crespocarlos marked this pull request as ready for review August 18, 2022 19:25
@crespocarlos crespocarlos requested a review from a team as a code owner August 18, 2022 19:25
Copy link
Contributor

@matschaffer matschaffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a couple code-level thoughts but mostly looks pretty good.

The test steps were pretty involved, but I was able to get some logs to show up.

Screen Shot 2022-08-24 at 14 57 02

If anyone else tries, be careful about which kibana. At first I thought it didn't work but I was on the elastic-package kibana (5601) not the PR kibana (5602).

// 'data_stream.type' will always be 'logs' for `logs-*` indices.
should: [
{ term: { 'service.type': 'elasticsearch' } },
{ term: { 'data_stream.type': 'logs' } },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic here has me a little concerned that this might break for standalone filebeat.

Should we maybe have a test that loads in es-archives for both filebeat-* and logs-* to ensure either can work (or even both to account for a transitional case?).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, we've initially discussed that but ended up making sure service.type exists in datastreams as well. see elastic/integrations#4033 (comment)

const indexPatterns = getNewIndexPatterns({
type,
config: getConfigWithCcs(true),
moduleType: 'elasticsearch',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of wanted to put moduleType in the TestTuple, but I can see the ecsLegacyOnly: true. That got me wondering what ecsLegacyOnly: true/false is even really for, but that's beyond the scope of this PR I think.

x-pack/plugins/monitoring/server/lib/logs/detect_reason.ts Outdated Show resolved Hide resolved
@klacabane
Copy link
Contributor

@elasticmachine merge upstream

@klacabane klacabane added release_note:enhancement v8.5.0 Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services labels Aug 24, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

Copy link
Contributor

@matschaffer matschaffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I'd say this is looking pretty good, only thing I'd change before merge is that inaccurate comment.

x-pack/plugins/monitoring/server/lib/logs/get_logs.ts Outdated Show resolved Hide resolved
config: getConfigWithCcs(false),
moduleType: 'elasticsearch',
});
expect(indexPatterns).toBe(expected);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repetition in these test cases still makes me twitch a little, but maybe best left for future enhancement.

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @crespocarlos @klacabane

@klacabane klacabane merged commit 1a82f0d into elastic:main Aug 26, 2022
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Aug 26, 2022
Mpdreamz pushed a commit to Mpdreamz/kibana that referenced this pull request Sep 6, 2022
…139121)

* Add log-* index pattern support on SM UI

* logs query fix

* remove datastream dedicated filter

* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'

* remove comment

* renaming filebeat leftovers to logs

* extract elasticsearch logs filter

* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'

* update filter

* typo

Co-authored-by: klacabane <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting release_note:enhancement Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v8.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Stack Monitoring] update index patterns with filebeat-* to use integration logs-*
6 participants