[Security Solution] Implement rule monitoring dashboard

[banderror]

Epic: https://github.com/elastic/security-team/issues/6032 (internal)

Summary

This PR adds a new [Elastic Security] Detection rule monitoring Kibana dashboard and a new POST /internal/detection_engine/health/_setup API endpoint.

Dashboard

The dashboard can be helpful for monitoring the health and performance of Security detection rules. Users of the dashboard must have read access to the .kibana-event-log-* index (see below for exact requirements). The dashboard is automatically installed into the current Kibana space when a user visits a page in Security Solution - similar to how we install the Fleet package with prebuilt detection rules.

API endpoint

The PR also adds a new endpoint for setting up anything related to monitoring rules and the health of the Detection Engine. If you call the endpoint, it will install the new dashboard to the Default Kibana space:

POST /internal/detection_engine/health/_setup

In order to install the dashboard to a different Kibana space, you will need to call it like that:

POST /s/<space-id>/internal/detection_engine/health/_setup

RBAC requirements

For installing the dashboard

The user calling the POST /internal/detection_engine/health/_setup endpoint must have at least Read access to Security Solution. No additional privileges are required, because the endpoint installs the dashboard on behalf of the internal user (kibana_system).

For viewing the dashboard

In order to be able to view the dashboard, users will need to have at least the following 2 privileges:

1. At least Read access to Kibana dashboards (Analytics -> Dashboard : Read)

2. At least read access to the .kibana-event-log-* index.

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
  • [DOCS] Document the new Detection rule monitoring dashboard security-docs#3478
• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[jpdjere]

Installed successfully, tested and working as described 👍

Are we not planning on adding this dashboard to the Security Dashboards page?
It's hard to find through the general dashboards list.

[approksiu]

Installed successfully, tested and working as described 👍

Are we not planning on adding this dashboard to the Security Dashboards page? It's hard to find through the general dashboards list.

We should add it there - by adding "Security Solution" tag. cc @banderror

[banderror]

Based on the received feedback I'm starting to work on the following fixes and improvements:

• Docs
  • Open a ticket for documenting the dashboard in the user-facing docs (ticket)
  • Known issue should be added to the release notes: rule can write multiple final statuses per execution; can be often caused by gap detection (ticket)
• Fix the dashboard itself
  • Change the "top 5 rules" label to "top 5 rules per interval"
  • Add descriptions to all the panels
  • Fix the 3 tables at the bottom by adding columns properly
  • Add the Managed and Security Solution tags to the dashboard as it was done for the Threat Intelligence Overview dashboard
• Fix the way the dashboard gets installed
  • Install it on behalf of the internal user
  • Remove the installation on Kibana server start
  • Call the _setup endpoint from all Security Solution pages, similar to how we install the package with prebuilt rules

[banderror]

@approksiu @jpdjere I addressed all the feedback and updated the PR description accordingly. Hopefully, the CI will pass and the PR will be merged soon.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 2afaa0f

Failed CI Steps

• FTR Configs #11

Test Failures

• [job] [logs] FTR Configs #11 / telemetry Telemetry service Opt-in/out banners shows the banner in the default configuration

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	4197	4198	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	10.9MB	11.0MB	+17.7KB

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	13	15	+2
securitySolution	411	415	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	14	16	+2
securitySolution	494	498	+4
total			+6

History

• 💚 Build #136358 succeeded 799686f
• 💛 Build #136346 was flaky c73c6ae6cd239ed1a160cae0e1e6ab91b1cdb10d
• 💔 Build #136327 failed 3016d42
• 💔 Build #136161 failed 50507b21b97dc11e423f91bc43396f4ed09f6a33
• 💔 Build #136051 failed 34f04a305e49f348766e246c200249ad527f92ad
• 💔 Build #135924 failed cd3c3586676957c2d3ba68a18250fb5cb142a96b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @banderror

[philippkahr]

@banderror don't know where to comment else wise. If you check the data view you are using you can append a data formatter for the fields. E.g. the duration fields, you can mark them as duration and milliseconds. This will change the output from numbers to readable output and automatically scale accordingly. All of the sudden you are in the 100s of seconds? You would see 100 seconds, or 1m 40s as a value.

[banderror]

Thanks for the suggestion @philippkahr, I think this could be a nice usability enhancement. Would you mind opening an issue in https://github.com/elastic/kibana for that?