[Security Solution] Detection rule migration mechanism

[banderror]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Blocker for: #184113
Related to: #50216

Summary

This is our own ticket to track the development of a mechanism for migrating rules in the Alerting Framework. The ResponseOps team tracks it in #50216.

In Security Solution, we can use this ticket to plan our own work on the mechanism, e.g. we can contribute to its design, propose any solutions, or open an RFC.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[marshallmain]

After discussing with @approksiu we see 3 potential new fields that we might want to add to rules soon (exact parameter names tbd):

• rule_source.repo: in Detections as Code we envision allowing users to connect external repositories of rules, either from VCS or other spaces or clusters, in addition to the Elastic prebuilt rules. This field would keep track of which external repository a rule is associated with. It's a net new field and we should be able to work around not having migrations by assuming that the repo is "elastic prebuilt rules" if a rule is immutable and the repo field is not populated.
• suppression.enabled: Some users have asked for prebuilt rules to come with suppression settings preconfigured, with a switch to just turn suppression on or off. An enabled flag for suppression would be a natural way to do this. We should be able to add the new enabled field and assume it is true if the field is not set.
• rule_lifecycle: A new metadata field for describing if the rule is in development, testing, prod, etc. As metadata this would be optional and not require any extra effort to workaround migrations.

As these 3 are net new fields and we do not see a near term use case where rule migrations are a hard blocker, I think it's appropriate to prioritize Alerting Scale++ over rule migrations. However, rule migrations would be useful for rule_source.repo and suppression.enabled so we can manage and reduce overall complexity by removing assumptions that we must make when the field is not present.

cc @kobelb - in summary, I'm comfortable giving rule migrations lower priority than Alerting Scale++.

[kobelb]

@marshallmain thank you for tracking down the product needs so quickly.