[on hold] [ResponseOps] change AAD indices to data streams

[pmuellr]

will probably go with this PR instead: #160572

resolves #154266

Summary

Change the alerts-as-data indices from ILM-backed aliases/indices to data streams.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: cd1d6de
• Interpreting CI Failures

Failed CI Steps

• Jest Tests #14
• FTR Configs #2
• FTR Configs #2
• FTR Configs #4
• FTR Configs #4
• FTR Configs #16
• FTR Configs #16
• FTR Configs #17
• FTR Configs #17
• FTR Configs #18
• FTR Configs #18
• FTR Configs #19
• FTR Configs #19
• FTR Configs #22
• FTR Configs #22
• FTR Configs #30
• FTR Configs #30
• FTR Configs #34
• FTR Configs #34
• FTR Configs #42
• FTR Configs #42
• FTR Configs #47
• FTR Configs #47
• FTR Configs #49
• FTR Configs #49
• FTR Configs #52
• FTR Configs #52
• FTR Configs #54
• FTR Configs #54
• FTR Configs #55
• FTR Configs #55
• FTR Configs #58
• FTR Configs #58
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #1
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #1
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #2
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #2
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #3
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #3
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #4
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #4

Test Failures

• [job] [logs] FTR Configs #16 / Alerting alerts_as_data install alerts as data resources should install context specific alerts as data resources on startup
• [job] [logs] FTR Configs #16 / Alerting alerts_as_data install alerts as data resources should install context specific alerts as data resources on startup
• [job] [logs] FTR Configs #30 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts "after all" hook in "alerts"
• [job] [logs] FTR Configs #34 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts "after all" hook in "alerts"
• [job] [logs] FTR Configs #30 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts "after all" hook in "alerts"
• [job] [logs] FTR Configs #34 / alerting api integration security and spaces enabled - Group 2 Alerts alerts alerts "after all" hook in "alerts"
• [job] [logs] Jest Tests #14 / Alerts Service createAlertsClient() should create new AlertsClient
• [job] [logs] Jest Tests #14 / Alerts Service createAlertsClient() should return null if shouldWrite is false
• [job] [logs] Jest Tests #14 / Alerts Service register() does not create new index if concrete write index exists
• [job] [logs] Jest Tests #14 / Alerts Service register() does not updating settings or mappings if no existing concrete indices
• [job] [logs] Jest Tests #14 / Alerts Service register() should correctly install resources for context when common initialization is complete
• [job] [logs] Jest Tests #14 / Alerts Service register() should correctly install resources for context when secondaryAlias is defined
• [job] [logs] Jest Tests #14 / Alerts Service register() should correctly install resources for context when useEcs is true
• [job] [logs] Jest Tests #14 / Alerts Service register() should correctly install resources for context when useLegacyAlerts is true
• [job] [logs] Jest Tests #14 / Alerts Service register() should correctly install resources for custom namespace on demand when isSpaceAware is true
• [job] [logs] Jest Tests #14 / Alerts Service register() should log error and set initialized to false if checking for concrete write index throws error
• [job] [logs] Jest Tests #14 / Alerts Service register() should log error and set initialized to false if concrete indices exist but none are write index
• [job] [logs] Jest Tests #14 / Alerts Service register() should log error and set initialized to false if create concrete index throws error
• [job] [logs] Jest Tests #14 / Alerts Service register() should log error and set initialized to false if create concrete index throws resource_already_exists_exception error and write index does not already exists
• [job] [logs] Jest Tests #14 / Alerts Service register() should log error and set initialized to false if updating index mappings for existing indices throws error
• [job] [logs] Jest Tests #14 / Alerts Service register() should log error and set initialized to false if updating index settings for existing indices throws error
• [job] [logs] Jest Tests #14 / Alerts Service register() should not install component template for context if fieldMap is empty
• [job] [logs] Jest Tests #14 / Alerts Service register() should not throw error if checking for concrete write index throws 404
• [job] [logs] Jest Tests #14 / Alerts Service register() should not throw error if create concrete index throws resource_already_exists_exception error and write index already exists
• [job] [logs] Jest Tests #14 / Alerts Service register() should not update index template if simulating template throws error
• [job] [logs] Jest Tests #14 / Alerts Service register() should skip updating index mapping for existing indices if simulate index template throws error
• [job] [logs] FTR Configs #55 / cases security and spaces enabled: basic Common update_alert_status "before each" hook for "should update the status of multiple alerts attached to multiple cases using the cases client"
• [job] [logs] FTR Configs #55 / cases security and spaces enabled: basic Common update_alert_status "before each" hook for "should update the status of multiple alerts attached to multiple cases using the cases client"
• [job] [logs] FTR Configs #47 / cases security and spaces enabled: trial push_case memoryless server alerts "before each" hook for "should change the status of all alerts attached to a case to closed when closure_type: close-by-pushing and syncAlerts: true"
• [job] [logs] FTR Configs #47 / cases security and spaces enabled: trial push_case memoryless server alerts "before each" hook for "should change the status of all alerts attached to a case to closed when closure_type: close-by-pushing and syncAlerts: true"
• [job] [logs] Jest Tests #14 / createLifecycleExecutor set flapping on the document updates documents with flapping for active alerts
• [job] [logs] Jest Tests #14 / createLifecycleExecutor set flapping on the document updates existing documents for recovered alerts
• [job] [logs] Jest Tests #14 / createLifecycleExecutor set maintenance window ids on the document does not update documents with maintenance window ids for recovered alerts
• [job] [logs] Jest Tests #14 / createLifecycleExecutor set maintenance window ids on the document does not update documents with maintenance window ids for repeatedly firing alerts
• [job] [logs] Jest Tests #14 / createLifecycleExecutor set maintenance window ids on the document updates documents with maintenance window ids for newly firing alerts
• [job] [logs] Jest Tests #14 / createLifecycleExecutor updates existing documents for recovered alerts
• [job] [logs] Jest Tests #14 / createLifecycleExecutor updates existing documents for repeatedly firing alerts
• [job] [logs] Jest Tests #14 / createLifecycleExecutor writes initial documents for newly firing alerts
• [job] [logs] Jest Tests #14 / createLifecycleRuleTypeFactory with a new rule when alerts are new writes the correct alerts
• [job] [logs] Jest Tests #14 / createLifecycleRuleTypeFactory with a new rule when alerts recover writes the correct alerts
• [job] [logs] FTR Configs #2 / detection engine api basic license query_signals_route and find_alerts_route backwards compatibility "before all" hook for "should be able to filter old signals on host.os.name.caseless using runtime field"
• [job] [logs] FTR Configs #2 / detection engine api basic license query_signals_route and find_alerts_route backwards compatibility "before all" hook for "should be able to filter old signals on host.os.name.caseless using runtime field"
• [job] [logs] FTR Configs #16 / detection engine api security and spaces enabled - Group 1 create_index elastic admin with another index that shares index alias "before all" hook for "should report that signals index does not exist"
• [job] [logs] FTR Configs #16 / detection engine api security and spaces enabled - Group 1 create_index elastic admin with another index that shares index alias "before all" hook for "should report that signals index does not exist"
• [job] [logs] FTR Configs #19 / detection engine api security and spaces enabled - Group 10 Signals migration status "before each" hook for "returns no indexes if no signals exist in the specified range"
• [job] [logs] FTR Configs #19 / detection engine api security and spaces enabled - Group 10 Signals migration status "before each" hook for "returns no indexes if no signals exist in the specified range"
• [job] [logs] FTR Configs #49 / detection engine api security and spaces enabled - Group 6 Detection engine signals/alerts compatibility Alerts Compatibility CTI "before each" hook for "allows querying of legacy enriched signals by threat.indicator"
• [job] [logs] FTR Configs #49 / detection engine api security and spaces enabled - Group 6 Detection engine signals/alerts compatibility Alerts Compatibility CTI "before each" hook for "allows querying of legacy enriched signals by threat.indicator"
• [job] [logs] FTR Configs #58 / detection engine api security and spaces enabled - rule execution logic Query type rules "before all" hook for "should have the specific audit record for _id or none of these tests below will pass"
• [job] [logs] FTR Configs #58 / detection engine api security and spaces enabled - rule execution logic Query type rules "before all" hook for "should have the specific audit record for _id or none of these tests below will pass"
• [job] [logs] FTR Configs #54 / Dev Tools Search Profiler Editor No indices "before all" hook for "returns error if profile is executed with no valid indices"
• [job] [logs] FTR Configs #54 / Dev Tools Search Profiler Editor No indices "before all" hook for "returns error if profile is executed with no valid indices"
• [job] [logs] FTR Configs #17 / Endpoint plugin Resolver tests Resolver tests for the entity route signals index mapping tests "before all" hook for "returns an event even if it does not have a mapping for entity_id"
• [job] [logs] FTR Configs #17 / Endpoint plugin Resolver tests Resolver tests for the entity route signals index mapping tests "before all" hook for "returns an event even if it does not have a mapping for entity_id"
• [job] [logs] FTR Configs #42 / lens app - group 6 lens no data "before all" hook for "when no data opens integrations"
• [job] [logs] FTR Configs #42 / lens app - group 6 lens no data "before all" hook for "when no data opens integrations"
• [job] [logs] FTR Configs #55 / MetricsUI Endpoints Metric threshold rule > alert and action creation rule should be active
• [job] [logs] FTR Configs #4 / ObservabilityApp Observability alerts > "before all" hook in "Observability alerts >"
• [job] [logs] FTR Configs #4 / ObservabilityApp Observability alerts > "before all" hook in "Observability alerts >"
• [job] [logs] Jest Tests #14 / resourceInstaller if write is enabled should install namespace level resources for the default space
• [job] [logs] Jest Tests #14 / resourceInstaller updateAliasWriteIndexMapping() gracefully fails on empty mappings
• [job] [logs] Jest Tests #14 / resourceInstaller updateAliasWriteIndexMapping() gracefully fails on error simulating mappings
• [job] [logs] Jest Tests #14 / resourceInstaller updateAliasWriteIndexMapping() succeeds on the happy path
• [job] [logs] Jest Tests #14 / RuleDataClient bulk() waits until cluster client is ready before calling bulk
• [job] [logs] Jest Tests #14 / RuleDataClient getReader() getReader searchs an index pattern without a wildcard when the namespace is provided
• [job] [logs] Jest Tests #14 / RuleDataClient getReader() waits until cluster client is ready before searching
• [job] [logs] FTR Configs #18 / rules security and spaces enabled: basic ruleRegistryAlertsSearchStrategy logs "before all" hook for "should return alerts from log rules"
• [job] [logs] FTR Configs #18 / rules security and spaces enabled: basic ruleRegistryAlertsSearchStrategy logs "before all" hook for "should return alerts from log rules"
• [job] [logs] FTR Configs #52 / Session View API (basic) Session view - /internal/session_view/process_events - with a basic license using typical process event data "before all" hook for "/internal/session_view/process_events returns a page of process events"
• [job] [logs] FTR Configs #52 / Session View API (basic) Session view - /internal/session_view/process_events - with a basic license using typical process event data "before all" hook for "/internal/session_view/process_events returns a page of process events"
• [job] [logs] FTR Configs #22 / Triggers Actions UI Example Rule status dropdown "before all" hook for "should load from the shareable lazy loader"
• [job] [logs] FTR Configs #22 / Triggers Actions UI Example Rule status dropdown "before all" hook for "should load from the shareable lazy loader"

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	413	417	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	497	501	+4
total			+6

History

• 💔 Build #124630 failed b0dc243e51a77d23b1add5ea36227416240ab891
• 💔 Build #124424 failed afffba3184d7971d2c29d4de0891331ea1186b8c
• 💔 Build #124137 failed eb5f4f8e637da8673f81bd0beafb95cc55bc56a1

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[pmuellr]

closing as we merged the real thing in #160572