[ResponseOps] change event log to use a datastream

[pmuellr]

Summary

closing this in favor of #154664 (but I may change my mind)

Changes event log from using indices, aliases, and ILM manually, to using datastreams.

Partially resolving #154266

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: 770acdf
• Interpreting CI Failures

Failed CI Steps

• Build API Docs
• Jest Tests #13
• x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts
• x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts
• x-pack/test/functional/apps/lens/group3/config.ts
• x-pack/test/functional/apps/lens/group3/config.ts
• FTR Configs #4
• FTR Configs #4
• FTR Configs #6
• FTR Configs #6
• FTR Configs #9
• FTR Configs #9
• FTR Configs #15
• FTR Configs #17
• FTR Configs #17
• FTR Configs #30
• FTR Configs #30
• FTR Configs #31
• FTR Configs #31
• FTR Configs #43
• Check Types

Test Failures

• [job] [logs] FTR Configs #31 / Actions schedule unsecured action should successfully schedule email action
• [job] [logs] FTR Configs #31 / Actions schedule unsecured action should successfully schedule email action
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerting and Actions Telemetry telemetry should retrieve telemetry data in the expected format
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerting and Actions Telemetry telemetry should retrieve telemetry data in the expected format
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule actions when appropriate
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all at space1 should schedule task, run alert and schedule actions when appropriate
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all_with_restricted_fixture at space1 should schedule task, run alert and schedule actions when appropriate
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts space_1_all_with_restricted_fixture at space1 should schedule task, run alert and schedule actions when appropriate
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule actions when appropriate
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts superuser at space1 should schedule task, run alert and schedule actions when appropriate
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts eventLog should generate events for alert decrypt errors
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts eventLog should generate events for alert decrypt errors
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts excluded should handle create alert request appropriately
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts excluded should handle create alert request appropriately
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts getActionErrorLog gets action error logs from an alternate space
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts getActionErrorLog gets action error logs from an alternate space
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts getGlobalExecutionKpi should return KPI from all spaces in the namespaces param
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts getGlobalExecutionKpi should return KPI from all spaces in the namespaces param
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts getGlobalExecutionKpi should return KPI only from the current space
• [job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group2/config.ts / alerting api integration security and spaces enabled - Group 2 Alerts alerts getGlobalExecutionKpi should return KPI only from the current space
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes cancellable_rule rule that implements cancellation services runs successfully if it does not timeout
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes cancellable_rule rule that implements cancellation services runs successfully if it does not timeout
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes cancellable_rule rule that implements cancellation services throws an error if execution is short circuited
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes cancellable_rule rule that implements cancellation services throws an error if execution is short circuited
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes cancellable_rule rule that implements cancellation services throws an error if search runs longer than rule timeout
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes cancellable_rule rule that implements cancellation services throws an error if search runs longer than rule timeout
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers index threshold rule that hits max alerts circuit breaker persist existing alerts to next execution if circuit breaker is hit
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers index threshold rule that hits max alerts circuit breaker persist existing alerts to next execution if circuit breaker is hit
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers rule that hits max alerts circuit breaker completes execution if rule type requests alert limit and reports back whether it reached the limit
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers rule that hits max alerts circuit breaker completes execution if rule type requests alert limit and reports back whether it reached the limit
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers rule that hits max alerts circuit breaker ends in error if rule type requests alert limit but does not report back whether it reached the limit
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers rule that hits max alerts circuit breaker ends in error if rule type requests alert limit but does not report back whether it reached the limit
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers rule that hits max alerts circuit breaker short circuits rule execution if rule type lets framework handle alert limit
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes circuit_breakers rule that hits max alerts circuit breaker short circuits rule execution if rule type lets framework handle alert limit
• [job] [logs] FTR Configs #4 / Alerting builtin alertTypes index_threshold rule runs and gracefully handles ES errors
• [job] [logs] FTR Configs #4 / Alerting builtin alertTypes index_threshold rule runs and gracefully handles ES errors
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes long_running_rule long running rule still logs alert docs when rule exceeds timeout when cancelAlertsOnRuleTimeout is false on rule type
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes long_running_rule long running rule still logs alert docs when rule exceeds timeout when cancelAlertsOnRuleTimeout is false on rule type
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes long_running_rule long running rule writes event log document for timeout for each rule execution that ends in timeout - every execution times out
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes long_running_rule long running rule writes event log document for timeout for each rule execution that ends in timeout - every execution times out
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes long_running_rule long running rule writes event log document for timeout for each rule execution that ends in timeout - some executions times out
• [job] [logs] FTR Configs #6 / Alerting builtin alertTypes long_running_rule long running rule writes event log document for timeout for each rule execution that ends in timeout - some executions times out
• [job] [logs] FTR Configs #6 / Alerting Capped action type should not trigger actions more than connector types limit
• [job] [logs] FTR Configs #6 / Alerting Capped action type should not trigger actions more than connector types limit
• [job] [logs] FTR Configs #6 / Alerting ephemeral should execute all requests, when some will be ephemeral and some not
• [job] [logs] FTR Configs #6 / Alerting ephemeral should execute all requests, when some will be ephemeral and some not
• [job] [logs] FTR Configs #6 / Alerting eventLog alerts should generate expected alert events for normal operation
• [job] [logs] FTR Configs #6 / Alerting eventLog alerts should generate expected alert events for normal operation
• [job] [logs] FTR Configs #17 / Alerting monitoring_collection inMemoryMetrics should count timeouts
• [job] [logs] FTR Configs #17 / Alerting monitoring_collection inMemoryMetrics should count timeouts
• [job] [logs] FTR Configs #6 / Alerting notifyWhen alert with notifyWhen=onActionGroupChange should execute actions when action group changes
• [job] [logs] FTR Configs #6 / Alerting notifyWhen alert with notifyWhen=onActionGroupChange should execute actions when action group changes
• [job] [logs] FTR Configs #6 / Alerting notifyWhen alert with notifyWhen=onActiveAlert should always execute actions
• [job] [logs] FTR Configs #6 / Alerting notifyWhen alert with notifyWhen=onActiveAlert should always execute actions
• [job] [logs] FTR Configs #6 / Alerting snooze should not trigger actions when snoozed
• [job] [logs] FTR Configs #6 / Alerting snooze should not trigger actions when snoozed
• [job] [logs] Jest Tests #13 / buffering documents should handle lots of docs correctly with a delay in the bulk index
• [job] [logs] Jest Tests #13 / buffering documents should write buffered docs after buffer exceeded
• [job] [logs] Jest Tests #13 / buffering documents should write buffered docs after timeout
• [job] [logs] FTR Configs #15 / cases security and spaces enabled: trial push_case incident recorder server should format the comments correctly
• [job] [logs] Jest Tests #13 / createIndex should call cluster with proper arguments
• [job] [logs] Jest Tests #13 / createIndex should throw error when not getting an error of type resource_already_exists_exception
• [job] [logs] Jest Tests #13 / createIndex shouldn't throw when an error of type resource_already_exists_exception is thrown
• [job] [logs] FTR Configs #9 / detection engine api security and spaces enabled - Group 10 Get Rule Execution Results should return execution events for a rule that has executed in a failure state with a gap
• [job] [logs] FTR Configs #9 / detection engine api security and spaces enabled - Group 10 Get Rule Execution Results should return execution events for a rule that has executed in a failure state with a gap
• [job] [logs] FTR Configs #9 / detection engine api security and spaces enabled - Group 10 Get Rule Execution Results should return execution events ordered by @timestamp desc when a status filter is active and there are more than 1000 executions
• [job] [logs] FTR Configs #9 / detection engine api security and spaces enabled - Group 10 Get Rule Execution Results should return execution events ordered by @timestamp desc when a status filter is active and there are more than 1000 executions
• [job] [logs] FTR Configs #9 / Dev Tools Search Profiler Editor No indices "before all" hook for "returns error if profile is executed with no valid indices"
• [job] [logs] FTR Configs #9 / Dev Tools Search Profiler Editor No indices "before all" hook for "returns error if profile is executed with no valid indices"
• [job] [logs] Jest Tests #13 / doesAliasExist should call cluster with proper arguments
• [job] [logs] Jest Tests #13 / doesAliasExist should return false when call cluster returns false
• [job] [logs] Jest Tests #13 / doesAliasExist should return true when call cluster returns true
• [job] [logs] Jest Tests #13 / doesAliasExist should throw error when call cluster throws an error
• [job] [logs] Jest Tests #13 / indexDocument should call cluster client bulk with given doc
• [job] [logs] Jest Tests #13 / indexDocument should log an error when cluster client throws an error
• [job] [logs] Jest Tests #13 / initializeEs should continue initialization if updating existing index aliases throws an error
• [job] [logs] Jest Tests #13 / initializeEs should continue initialization if updating existing index settings throws an error
• [job] [logs] Jest Tests #13 / initializeEs should continue initialization if updating existing index templates throws an error
• [job] [logs] Jest Tests #13 / initializeEs should create index template if it doesn't exist
• [job] [logs] Jest Tests #13 / initializeEs should create initial index if it doesn't exist
• [job] [logs] Jest Tests #13 / initializeEs shouldn't create index template if it already exists
• [job] [logs] Jest Tests #13 / initializeEs shouldn't create initial index if it already exists
• [job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens no data "before all" hook for "when no data opens integrations"
• [job] [logs] x-pack/test/functional/apps/lens/group3/config.ts / lens app - group 3 lens no data "before all" hook for "when no data opens integrations"
• [job] [logs] Jest Tests #13 / shutdown() should work if some docs have been written

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[pmuellr]

closing (perhaps temporarily) in favor of #154664