[ResponseOps] change event log to use a datastream

[pmuellr]

Summary

resolves #62677

Changes event log from using indices, aliases, and ILM manually, to using data streams.

In general for the event log, we create a new set of indices / aliases / index template for each stack version. And then when we search over all event logs, we use a wildcard pattern to include all the existing versions. That continues with this PR. Just instead of creating indices / aliases for a new version, we create a data stream.

Existing event log indices / aliases are left alone, since the current way to search over existing event logs is with a wildcard. For instance, .kibana-event-log-8.7.0 is the alias for 8.7.0 and .kibana-event-log-8.9.0 will be the data stream for 8.9.0. Searches over the event log are done using the index pattern .kibana-event-log-*, so will search over all the aliases and data streams.

Rolling back versions isn't a problem - the "new" resources created on the migration to a newer version, would only ever be referenced by search, if the deployment was rolled back to a previous verison. The existing indices / aliases / data streams for that version are left intact, the same structure they were in the release they were created in.

To test

The basic test is to ensure the event log is stil operational. You can do by checking the history tab of a running rule, which works off the event log.

To make sure we can still see old events from the older aliases / indices, you should actually start with an older version, create a rule and start it running, note the time the rule was started. Then migrate that deployment to this PR. Once Kibana comes up, check that the rule history still includes the run from the previous verison.

Checklist

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials (updated README)
• Unit or functional tests were updated or added to match the most common scenarios

[pmuellr]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mikecote]

@pmuellr I found this issue #62677 and linked your PR to resolve it when merged.

[pmuellr]

Converted to a draft - shouldn't have taken it out of draft! :-)

I've got to add a scheme for this new datastream to co-exist with old-style event log indices.

[ymao1]

@pmuellr In the issue description for changing .alerts to datastream, it says

Serverless Elasticsearch will be replacing index lifecycle management with datastream lifecycle management, and we will no longer be allowed to create the .alerts-ilm-policy

Does this also apply to the event log? I see we are still manually creating an ILM policy for the event log

[pmuellr]

Does this also apply to the event log? I see we are still manually creating an ILM policy for the event log

Yes, it does, but we'll need it for other-than-serverless. My understanding is the data stream lifecycle management isn't quite done yet. Once it is, then I think the next task would be to customize startup for severless to do whatever we need to do for the ILM-replacement thing. Does that sound right @kobelb ?

[kobelb]

Yes, it does, but we'll need it for other-than-serverless. My understanding is the data stream lifecycle management isn't quite done yet. Once it is, then I think the next task would be to customize startup for severless to do whatever we need to do for the ILM-replacement thing. Does that sound right @kobelb ?

Yup!

[ymao1]

LGTM! Verified that a datastream is created for 8.9 and that the exec log could read from both the data stream and pre-8.9 event log indices.

[pmuellr]

@elasticmachine merge upstream

[pmuellr]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 9ce7148
• Cloud Deployment

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	413	417	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	497	501	+4
total			+6

History

• 💚 Build #132864 succeeded ecef68c
• 💛 Build #130201 was flaky b6a1b80
• 💔 Build #130071 failed c9191c3ad539b428874fb6144cb2221fd3e96082
• 💛 Build #129757 was flaky 3da45d3cc7b7cb3ad7736f3f2e8707c441e7ec7a
• 💛 Build #129679 was flaky ef31ed4892ff26bc4638fa9edeac6d4003fd1afa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pmuellr

[spong]

Security Solution integration test update LGTM! Thank you for this event-log upgrade @pmuellr!