[Discussion][Security Solution] Enforcing unique rule_id's within Integrations' security-rule Saved Objects

[spong]

Summary

With the introduction of the LotL Attack Detection package (elastic/integrations#2115), we'll now have two packages that are bundling Security Detection Rule assets that can be installed within the Security Solution App (the other being the Prebuilt Security Detection Rules package).

One constraint of Security Detection Rule's is that they each must contain a unique rule_id, an identifier that can be thought of as an external id that remains static as the asset traverses different systems, or is exported/re-imported, etc. This rule_id (along with the SO id) must remain unique per Kibana space. If a rule is duplicated, a new rule_id is generated. This rule_id also aids in features like displaying all alerts that a rule has ever had, e.g. on the Rule Details page we query for alerts by this id and can show all alerts from past instantiations of the Rule in cases where it was exported, deleted, re-imported, etc.

Open Question

With the introduction of a second package containing security-rule assets, and with more to come, the open question for discussion here is how do we ensure unique rule_id's across all packages shipping Security Detection Rules.

Impact

While the impact here is high (where two assets with the same rule_id will result in only one installing within the Security App, with no error to the user), the actual chance of this happening is pretty low, and will most likely be the result of developer error when creating/updating packages, or developers using a common string instead of uuid.v4(). E.g. creating a new package and copying over existing assets as an example and forgetting to update rule_id, or perhaps as a byproduct from the central management of the assets before being added to the package (something which the detection-rules repo checks for).

Potential Solutions

Ideally, we'd just have a test that would run within the integrations repo and it would check for any duplicates across all packages before merging, however this fails to take into account that not all packages are hosted in the integrations repo, and that there are packages whose source may be in other repositories (elastic/integrations#2115 (comment)).

So that said, here are a few potential solutions and corresponding changes in UX that users/developers may incur:

1. Prefix security-rule(Fleet SO)/detection-rule(Kibana SO) rule_id's with package name

This will help ensure uniqueness between packages, but has the following cons:

• Would be a breaking changing to the existing Prebuilt Security Detection Rules package rules (or those would need to be coded as an exception),
• It introduces complexity if multiple packages want to ship the same rule (e.g. when we start including rules in their corresponding integration)
• Is just a process change, and so could still result in the same rule_id being used within the same package (or would it be possible to add tests before a package can be published to check this @jsoriano?)
• Leaks implementation data (where it came from) to a field that's intended to behave as a universal external id (though this could go either way)

2. Add package name to security-rule/detection-rule assets

• Would require additional pre-install/upgrade logic on the Security Solution side to ensure rule_id uniqueness, and introduces added complexity when importing rules (now rule_id wouldn't be static as-is, but would follow special business logic depending on certain fields)
• Would need to be differentiated from any package fields added to indicate dependency on source data (i.e. this package collects the data necessary for this rule vs this package bundles this rule)
• Tough to guarantee 3rd party rule overlap (anyone could use this field even if it's not their own package)

3. Display error in UI when installing rules and allow user to pick which to install

• Doesn't catch errors till runtime

4. Add package/integration process docs to ensure developer verifies uniqueness

• Similar to 1. it's just a process change (and so could be ignored/missed), but we could supply a CLI tool or something they could run to verify ID's against a currently deployed package-registry

5. Do Nothing

• Although with an expected low probability of occurring, users could install packages and end up not seeing them in the Security UI and have no errors to point to as to why they can't find a specific rule from a package.

Please edit this comment and add any additional thoughts/solutions to this list! 🙂

Suggested Solution

Having just typed all these up 😅, I would propose a combination of solutions 3./4., where we add the appropriate error UI so users can be identified in the unlikely event there is an issue, give them the option to pick which rule to install, and let them know which package owner(s) to reach out to so they can address the issue, and then also add additional package/integration process docs that would support developers creating packages as to further decrease the likelihood of this even occurring in the first place.

cc @jethr0null @XavierM @elastic/security-detections-response-rules @alvarezmelissa87 @brokensound77

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/fleet (Feature:EPM)

[jsoriano]

Thanks @spong for the write-up.

the actual chance of this happening is pretty low, and will most likely be the result of developer error when creating/updating packages, or developers using a common string instead of uuid.v4(). E.g. creating a new package and copying over existing assets as an example and forgetting to update rule_id, or perhaps as a byproduct from the central management of the assets before being added to the package

This is not so uncommon under some circumstances. For example we found this kind of issues quite frequently in early versions of Fleet when migrating assets from Beats to integrations. Users of Beats found id conflicts when trying to use integrations in the same cluster. This may happen here too if security rules are migrated from an existing solution without changing ids as part of the process.

1. Prefix security-rule(Fleet SO)/detection-rule(Kibana SO) rule_id's with package name

• Is just a process change, and so could still result in the same rule_id being used within the same package (or would it be possible to add tests before a package can be published to check this @jsoriano?)

This can be enforced in the package spec, we have a similar check for kibana dashboards, where we check that their ids are prefixed with the package name. We could do something similar here. Package spec validation is run during CI at least in packages whose source is in the integrations repo. And for all packages before publishing them.

Regarding the proposed solutions, I think that something like point 3 should be implemented in any case. Trying to install an asset with the same id as an existing one shouldn't fail transparently (neither should it overwrite the previously existing asset without telling anything to the user).

Point 4 is not very practical. Take into account that sooner or later, the public package registry is going to contain several thousands of packages and versions, for such a check you may need to download all these packages and check their contents, producing quite an overhead in the development process.

I think that if possible to have ids prefixed with the package name we should go for option 1, as this is a known solution we are also applying to other resources, and this is something we can check when publishing any package independently of the development process used by the maintainers of the package. We will need to see how to handle existing packages.
In any case something like option 3 should be also implemented, to prevent unexpected behaviours.

[spong]

Thank you for clarifying some of my assumptions above @jsoriano! 🙂

This is not so uncommon under some circumstances. For example we found this kind of issues quite frequently in early versions of Fleet when migrating assets from Beats to integrations. Users of Beats found id conflicts when trying to use integrations in the same cluster. This may happen here too if security rules are migrated from an existing solution without changing ids as part of the process.

This is good to know, thank you! This scenarios makes sense and would still be applicable to security, although I imagine more-so for 3rd party integration developers than internally since we centrally manage most security rules. That said, this will definitely increase in scope if other solutions adopt this same way of delivering pre-packaged rules via integrations.

This can be enforced in the package spec, we have a similar check for kibana dashboards, where we check that their ids are prefixed with the package name. We could do something similar here. Package spec validation is run during CI at least in packages whose source is in the integrations repo. And for all packages before publishing them.

Perfect! Will add this to the proposed solutions as the extra coverage here will at least protect an individual package from duplicating their own rule_id's internally.

Point 4 is not very practical. Take into account that sooner or later, the public package registry is going to contain several thousands of packages and versions, for such a check you may need to download all these packages and check their contents, producing quite an overhead in the development process.

Yeah, I was thinking more-so a dev documentation update so 3rd party devs following along could follow how they can correctly bundle security rules with their integration. Definitely makes sense why CLI tooling wouldn't work here 👍

[spong]

Alrighty! So from our discussion today we ended up with a mish-mash of the proposed solutions from above. To summarize though:

• We decided it was best to add another field to the security-rule SO like sourcePackage ([Security Solution][Detections] Update ruleAssetType Saved Object to include package information #128544) and we would then use this field along with rule_id for managing conflicts within the Security Solution app itself. Prefixing package name with rule_id was determined to be more restrictive for the intent of rule_id (an external system id), and would require an exception for all the existing prebuilt security rules (which we one day want to start shipping alongside their appropriate package anyway).

• We'll also be adding better error handling on the Security Solution side ([Security Solution][Detections] Support error scenario for rule_id conflicts between pre-packaged rules #128542) and try to inform users before they upgrade their rules that there could be potential conflicts, and how/what they can do to resolve them (most likely choosing one within the UI, and providing package information so they can report the issue to the package owner).

• Lastly, we'd like to introduce two tests within the integrations repo ([Security Solution][Detections] Add package tests for rule_id uniqueness and ensuring correct sourcePackage #128547) to ensure the security-rule sourcePackage field is the same name of the package it exists in, and that rule_id is at least unique between all the other security-rule assets in that package. @jsoriano, I created the aforementioned ticket on the kibana side to track and imagine our team would be doing this work, but if you could provide direction and/or a contact that could when it's time to schedule this work that would be great! (And of course any specific feedback or limitations we may have with this proposed path forward as well :)

And lastly lastly, it was agreed none of these changes would need to happen immediately (since we've verified unique ID's across the existing packages shipping security-rule assets) and that it would be sufficient to make these changes at some point in the near-term, and that the LotL and Prebuilt Detection Rules packages could just be updated to include the sourcePackage field at that time. So no reason to hold up elastic/integrations#2115 anymore than we have -- thank you @alvarezmelissa87 for your patience here!

If I have mis-conveyed any of the above, or if there are any further questions please comment here. I'll keep this issue open for a little while longer for this feedback, and if we're all good will go ahead and close this in favor of the three issues outlined above. Thank you everyone for you valuable feedback and input here! 🙂

[mtojek]

Lastly, we'd like to introduce two tests within the integrations repo (#128547) to ensure the security-rule sourcePackage field is the same name of the package it exists in, and that rule_id is at least unique between all the other security-rule assets in that package. @jsoriano, I created the aforementioned ticket on the kibana side to track and imagine our team would be doing this work, but if you could provide direction and/or a contact that could when it's time to schedule this work that would be great! (And of course any specific feedback or limitations we may have with this proposed path forward as well :)

@spong

On behalf of the Integrations Ecosystem team responsible for tooling and Package Storage maintenance, I must decline this idea due to multiple reasons:

1. Think Big. We are preparing our Elastic package universe to reach 1kk packages. This won't be possible with the single mono-repo model and most likely packages will land in different repositories. It won't be physically possible to check "everywhere" if a single rule ID doesn't cause a conflict.
2. Sources stored in the Integrations repository reflect only the latest revision of the package. Even if you test against all the latest revisions, it doesn't give you any warranty that your changes won't cause a conflict with the already published package.
3. Lessons learnt. We have already hit this problem with Kibana dashboard IDs and decided to prepend a package name an object ID (for example: apache_dashboard_overview, apache_access_logs_view, apache_ecfd3862-f5ea-46f7-a4f9-074ea0331cb6 etc.). This method doesn't require checking consistency across different packages, as the prefix enforces it. We have already introduced special handling for security rule. It would be great if we can revert it eventually and start treating them as other Kibana saved objects.

As we are code owners of the Integrations, Package Storage, Elastic Package, Package Spec, please keep us in the loop (@elastic/ecosystem).

FYI This thread fits better the Package Spec repository, which is the discussion point for all initiatives around packages and Fleet. Please consider moving it there. Thank you.

[spong]

We have already introduced special handling for security rule. It would be great if we can revert it eventually and start treating them as other Kibana saved objects.

There's what now?! 😮 🤯

Well then, that makes this whole thing a non-issue?? That code ensures rule_id is bound to id and 💥, we get our uniqueness check within fleet, at runtime, thanks to Kibana, no? Actually....after some testing it looks like integrations will just override any existing SO's with the same id....that doesn't seem good at all, which is why I presume you're starting to enforce prepending id's with the package name? I would expect the asset install to fail if there's already an SO with the same id (as it does if it's in the same asset package), so I'm curious why this would be the default behavior here? Seems like prefixing id's with the package name should be mandatory for any bundled kibana assets otherwise another package could come along and wipe out some other package's assets without any error to the user??

Of course my test environment may not be an accurate representation of how this is supposed to work, so sounds like it might be good for us to have a chat -- I'll reach out next week and we can discuss further 🙂

---

As an aside (and probably irrelevant now), in your response to the proposed solution, I believe there's a misunderstanding as the intent was not to check "everywhere" if a single rule ID doesn't cause a conflict (which @jsoriano pointed out as not feasible in elastic/integrations#2115 (comment)), but rather check only the assets included in an individual package to ensure their security-rule assets do not have any duplicate rule_id's between them. So in the case of the LoTL Attack Detection PR elastic/integrations#2115, the intent would be that package validation would fail if any of their security-rule assets (within packages/problemchild/kibana/security_rule/*.json) had any duplicate rule_id's -- essentially the exact same thing ValidateKibanaObjectIDs looks to be doing for id's.

---

FYI This thread fits better the Package Spec repository, which is the discussion point for all initiatives around packages and Fleet. Please consider moving it there. Thank you.

If it's okay I'm going to keep this in the Kibana repo until we have a chance to chat next week as I don't want to waste anymore of anyone's time until we can level-set on the facts here. Once we chat, if this still warrants an issue I'll go ahead and open a dedicated issue in the Package Spec repo with all the details distilled so folks don't have to go through any irrelevant/inaccurate information. 👍

[mtojek]

Actually....after some testing it looks like integrations will just override any existing SO's with the same id....that doesn't seem good at all, which is why I presume you're starting to enforce prepending id's with the package name?

Package validation according to the spec format is a procedure executed statically without the requirement of running Elasticsearch or Kibana. The procedure needs ~1s to check if the package is valid and scales well.

I would expect the asset install to fail if there's already an SO with the same id (as it does if it's in the same asset package), so I'm curious why this would be the default behavior here?

I will pass this question to @joshdover, but add my few cents. Such logic wouldn't be a proper validation mechanism, but rather an object conflict prevention method. It can't be the ultimate decision-making process as it's impossible to install ALL packages (consider the 1kk target).

[joshdover]

I would expect the asset install to fail if there's already an SO with the same id (as it does if it's in the same asset package), so I'm curious why this would be the default behavior here?

I can't really speak to the original intention when this code was written but it appears it's always worked this way. I know there is one case where we do want to overwrite SOs, which is when creating the logs-* and metrics-* data views, however that's a unique case that doesn't really apply here.

One question I have is: why do we need IDs in the SOs defined in packages at all? There are two use cases cases I'm aware of where this is depend upon:

• Objects that have references to other objects (eg. dashboards that reference visualizations).
• UIs in Kibana that reference / link to a specific object from a package

We definitely need to support these use cases, but is there an actual need for the ID in the package to actually match the ID that we end up creating the objects with at install time? Would it be acceptable to re-generate IDs when we install SOs and then provide an API in Kibana for translating the ID defined in the package to the ID the object was installed with?

This rule_id (along with the SO id) must remain unique per Kibana space.

As of 8.0, all Saved Objects now have globally unique IDs so I don't believe we need to consider this explicitly. Fleet does not yet really support installing package SOs into multiple spaces, but the idea in the future is that we would allow users to share the same SO across multiple spaces. We're tracking this as part of #99116 but are blocked on waiting for Dashboards to become share-capable.

[mtojek]

Objects that have references to other objects (eg. dashboards that reference visualizations).
UIs in Kibana that reference / link to a specific object from a package

I can confirm the above reasons. Not sure if there are any other, but we learnt about these the "hard way", breaking product :)

Would it be acceptable to re-generate IDs when we install SOs and then provide an API in Kibana for translating the ID defined in the package to the ID the object was installed with?

I expect that the SO format will change then, right? If we can ensure BWC with the older package, and the new translation logic will be aware of legacy packages, then I guess that we're good to go.

[spong]

Thanks for the context/history @joshdover & @mtojek!

We definitely need to support these use cases, but is there an actual need for the ID in the package to actually match the ID that we end up creating the objects with at install time? Would it be acceptable to re-generate IDs when we install SOs and then provide an API in Kibana for translating the ID defined in the package to the ID the object was installed with?

So on the Security Solution side (Detections specifically, as I can't speak for Endpoint), we don't have a direct need for specifying the ID unless we're managing object relations like you mention (Rules that reference Actions or Exception Lists or Timeline Templates, etc), so something like this should work fine for us! 👍

---

That said, I brought this up with the team in our weekly today and in effort to provide a holistic solution around Pre-packaged Rule Assets (not specific to just Security Solution, so we can hopefully remove that special handling you mentioned @mtojek ) @XavierM on the ResponseOps side is going to schedule a meet-and-greet for us all (ResponseOps, Security, O11y, and you folks on the Fleet/Ecosystem side) so we can touch base, meet the parties at play, and go from there.

[mtojek]

👍

not specific to just Security Solution, so we can hopefully remove that special handling you mentioned @mtojek

It may not be so simple to perform, because we need to ensure backward compatibility with older stacks. If you remove this rule from spec and developers start following it as "there is no special handling anymore", there is a risk that somebody will accidentally break the installation on an older stack.

To fully enable this option, we may need to release a major spec release and validate assets format depending on compatible Kibana stacks (package version constraints). If we want to follow that path, it's another internal discussion for the spec committee.