[Detections][EQL] EQL rule execution in detection engine #77419

marshallmain · 2020-09-14T22:47:28Z

Summary

Adds backend functionality for EQL rules in detection engine. Now updated to create a building block alert for each event within a sequence as well as an alert for the sequence as a whole. Non-sequence EQL queries work very similarly to KQL detection rules.

Chart of example sequence signal structure updated with building blocks

Work for follow up PRs:

Enrich shell alert by finding common field values across all events in a sequence
Add field for Detection Engine run instance (each time alert executor runs gets a unique id)
Add a schema version check before rule execution and if there is a version mis-match we'll throw an error and not create alerts, and then have documentation that references this error so users can search to find out how to upgrade their signals index
Add rule overrides for single event EQL queries

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-09-22T07:06:45Z

Pinging @elastic/endpoint-response (Team:Endpoint Response)

andrew-goldstein

Thanks for this enhancement @marshallmain! 🚀

Per the demo over zoom with @spong @peluja1012 and @XavierM, this PR LGTM WRT enabling the Investigate in timeline workflow, which should display (in Timeline) both the 🐢 "shell" detection, and the building block alerts representing the EQL sequences that triggered the shell detection.

More specifically, we validated that when a user clicks the Investigate in timeline action on a shell alert in Detections page, we can display both the shell and it's associated building blocks alerts in Timeline with the following (pseudo) query:

_id: id_of_the_shell_detection OR signal.group.id: id_of_the_shell_detection

marshallmain · 2020-09-24T15:23:36Z

@elasticmachine merge upstream

elasticmachine · 2020-09-24T20:20:12Z

Pinging @elastic/siem (Team:SIEM)

spong · 2020-09-24T21:06:58Z

Testing the field overrides with both sequence and non-sequence queries and doesn't look like they're working. No need to fix in this PR, but just commenting for tracking purposes. Not sure what we want to do here with sequence queries as there's no single source event (unless we want to use the shell alert with the common fields?), but queries which result in an alert created from a single event should be able to work with the existing override logic we have.

edit: Ahh yeah, I see in code now where you worked around the overrides... 🙂 This seems good to me for sequences, but shouldn't we be able to use the overrides for non-sequence results where one event results in one alert?

spong · 2020-09-25T00:27:29Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_rule.ts

@@ -65,7 +65,7 @@ export const buildRule = ({

  const meta = { ...ruleParams.meta, ...riskScoreMeta, ...severityMeta, ...ruleNameMeta };

-  const rule = pickBy<RulesSchema>((value: unknown) => value != null, {


I believe this is the last guard in place to remove any extraneous null/undefined's before writing the rule to the signals. In testing everything looks 👌 without (and I believe we've got tests covering this as well), but just want to verify the intent here.

Yeah converting the entire rule to Partial makes it harder to deal with when building the rest of the signal document since we need signal.rule.id later to compute the doc id. My thought is that we should be able to rely on the elasticsearch js library not to convert undefined fields to null or anything else so ES should never see undefined fields. Explicit null fields I think we should handle (usually reject) at the schema level instead of stripping them out later on. Some nulls are making it into the signal document since they're explicit in the rule SO.

I did more testing on this to figure out where the explicit null fields are coming from and we have some mismatches in the types between signalSchema and RuleTypeParams. For example, timestampOverride is schema.nullable(schema.string()) (so string | null) in signalSchema, but string | undefined in RuleTypeParams. So what happens when we don't define timestampOverride when creating a rule is it starts as undefined in the request but then the alert plugin validates using signalSchema which coerces the undefined into null and stores it in the SO. Then, when we get the rule SO back later to use in the alert executor, timestampOverride is string | null but RuleTypeParams says it's string | undefined and we use it as such. When it's null (which is when we originally sent undefined as part of rule creation), we end up with signal.rule.timestamp_override: null in the signal document.

The UI isn't displaying the null fields which is probably good. I think the best solution is to resolve the schema mismatches so that our undefined values stay undefined instead of being coerced into null.

Thanks for the extra sleuthing here! 🙂 And yeah, that was my worry with regards to the null's leaking into the signals. ++ to solving this via the schema mismatches to prevent that coercion in the first place.

spong

Checked out, tested locally and LGTM! Great clean code here @marshallmain, and tests to boot! 🙂

Congrats on all your hard work here to realize EQL within the Detection Engine, this is going to be an absolute game changer for our users! 🚀

marshallmain · 2020-09-25T13:55:29Z

@elasticmachine merge upstream

kibanamachine · 2020-09-25T15:27:50Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 9a6e908

Metrics [docs]

async chunks size

id	value	diff	baseline
securitySolution	10.2MB	+4.3KB	10.2MB

page load bundle size

id	value	diff	baseline
securitySolution	581.0KB	+136.0B	580.9KB

distributable file count

id	value	diff	baseline
default	45631	+1	45630

History

💔 Build #77034 failed 45ecb42
💚 Build #76982 succeeded 22383a9
💚 Build #76762 succeeded 1942bf9
💚 Build #76745 succeeded b42cf73
💔 Build #76553 failed f445d87

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

marshallmain · 2020-09-25T15:59:09Z

@spong yeah I was so focused on sequences the single event case was kind of an afterthought 😆 I'll add the overrides for those in the next PR if it's not a blocker here

* First draft of EQL rules in detection engine * Reorganize functions to separate files * Start adding eventCategoryOverride option for EQL rules * Add building block alerts for each event within sequence * Use eql instead of eql_query for rule type * Remove unused imports * Fix tests * Add basic tests for buildEqlSearchRequest * Add rulesSchema tests for eql * Add buildSignalFromSequence test * Add threat rule fields to buildRuleWithoutOverrides * Fix buildSignalFromSequence typecheck error * Add more tests * Add tests for wrapBuildingBlock and generateSignalId * Use isEqlRule function and fix import error * delete frank * Move sequence interface to types.ts * Fix import * Remove EQL execution placeholder, add back language to eql rule type * allow no indices for eql search * Fix unit tests for language update * Fix buildEqlSearchRequest tests * Replace signal.child with signal.group * remove unused import * Move sequence signal group building to separate testable function * Unbork the merge conflict resolution Co-authored-by: Elastic Machine <[email protected]>

…8550) * First draft of EQL rules in detection engine * Reorganize functions to separate files * Start adding eventCategoryOverride option for EQL rules * Add building block alerts for each event within sequence * Use eql instead of eql_query for rule type * Remove unused imports * Fix tests * Add basic tests for buildEqlSearchRequest * Add rulesSchema tests for eql * Add buildSignalFromSequence test * Add threat rule fields to buildRuleWithoutOverrides * Fix buildSignalFromSequence typecheck error * Add more tests * Add tests for wrapBuildingBlock and generateSignalId * Use isEqlRule function and fix import error * delete frank * Move sequence interface to types.ts * Fix import * Remove EQL execution placeholder, add back language to eql rule type * allow no indices for eql search * Fix unit tests for language update * Fix buildEqlSearchRequest tests * Replace signal.child with signal.group * remove unused import * Move sequence signal group building to separate testable function * Unbork the merge conflict resolution Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

@timestamp

* master: (226 commits) [Enterprise Search] Added Logic for the Credentials View (elastic#77626) [CSM] Js errors (elastic#77919) Add the @kbn/apm-config-loader package (elastic#77855) [Security Solution] Refactor useSelector (elastic#75297) Implement tagcloud renderer (elastic#77910) [APM] Alerting: Add global option to create all alert types (elastic#78151) [Ingest pipelines] Upload indexed document to test a pipeline (elastic#77939) TypeScript cleanup in visualizations plugin (elastic#78428) Lazy load metric & mardown visualizations (elastic#78391) [Detections][EQL] EQL rule execution in detection engine (elastic#77419) Update tutorial-full-experience.asciidoc (elastic#75836) Update tutorial-define-index.asciidoc (elastic#75754) Add support for runtime field types to mappings editor. (elastic#77420) [Monitoring] Usage collection (elastic#75878) [Docs][Actions] Add docs for Jira and IBM Resilient (elastic#78316) [Security Solution][Resolver] Update @timestamp formatting (elastic#78166) [Security Solution] Fix app layout (elastic#76668) [Security Solution][Resolver] 2 new functions to DAL (elastic#78477) Adds new elasticsearch client to telemetry plugin (elastic#78046) skip flaky suite (elastic#78512) (elastic#78511) (elastic#78510) (elastic#78509) (elastic#78508) (elastic#78507) (elastic#78506) (elastic#78505) (elastic#78504) (elastic#78503) (elastic#78502) (elastic#78501) (elastic#78500) ...

elasticmachine · 2021-09-23T14:26:00Z

Pinging @elastic/security-solution (Team: SecuritySolution)

marshallmain and others added 11 commits September 14, 2020 18:39

First draft of EQL rules in detection engine

6cf620f

Reorganize functions to separate files

82ae0c2

Start adding eventCategoryOverride option for EQL rules

9b450bf

Add building block alerts for each event within sequence

d3fce8a

Merge branch 'master' into eql-rules

ad9fe1b

Use eql instead of eql_query for rule type

922a21e

Remove unused imports

ecedb3f

Fix tests

3e45bc8

Add basic tests for buildEqlSearchRequest

4faa592

Add rulesSchema tests for eql

f9e26b0

Add buildSignalFromSequence test

3fd19f0

marshallmain marked this pull request as ready for review September 22, 2020 07:02

marshallmain requested review from a team as code owners September 22, 2020 07:02

marshallmain added v7.10.0 v8.0.0 release_note:enhancement Feature:Detection Rules Security Solution rules and Detection Engine Team:Endpoint Response Endpoint Response Team labels Sep 22, 2020

marshallmain requested review from peluja1012, FrankHassanabad, dhurley14, yctercero and rylnd September 22, 2020 14:22

marshallmain changed the title ~~[Detections] WIP First draft of EQL rules in detection engine~~ [Detections][EQL] EQL rule execution in detection engine Sep 22, 2020

marshallmain added 4 commits September 22, 2020 10:38

Add threat rule fields to buildRuleWithoutOverrides

bcdde62

Fix buildSignalFromSequence typecheck error

162256a

Add more tests

3f89fb9

Add tests for wrapBuildingBlock and generateSignalId

8b00aa7

marshallmain added 8 commits September 23, 2020 01:47

Fix import

5ebe577

Remove EQL execution placeholder, add back language to eql rule type

f445d87

allow no indices for eql search

1c5afd9

Fix unit tests for language update

506a3d6

Fix buildEqlSearchRequest tests

aacc605

Replace signal.child with signal.group

19cf9e2

remove unused import

b42cf73

Move sequence signal group building to separate testable function

1942bf9

andrew-goldstein approved these changes Sep 24, 2020

View reviewed changes

elasticmachine and others added 3 commits September 24, 2020 11:23

Merge branch 'master' into eql-rules

22383a9

Merge branch 'master' into eql-rules

8142b11

Unbork the merge conflict resolution

45ecb42

spong added the Team:SIEM label Sep 24, 2020

spong reviewed Sep 25, 2020

View reviewed changes

spong approved these changes Sep 25, 2020

View reviewed changes

Merge branch 'master' into eql-rules

9a6e908

marshallmain merged commit faf4b04 into elastic:master Sep 25, 2020

marshallmain deleted the eql-rules branch September 25, 2020 20:23

marshallmain mentioned this pull request Sep 25, 2020

[7.x] [Detections][EQL] EQL rule execution in detection engine (#77419) #78550

Merged

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detections][EQL] EQL rule execution in detection engine #77419

[Detections][EQL] EQL rule execution in detection engine #77419

marshallmain commented Sep 14, 2020 •

edited

Loading

elasticmachine commented Sep 22, 2020

andrew-goldstein left a comment

marshallmain commented Sep 24, 2020

elasticmachine commented Sep 24, 2020

spong commented Sep 24, 2020 •

edited

Loading

spong Sep 25, 2020

marshallmain Sep 25, 2020 •

edited

Loading

spong Sep 25, 2020

spong left a comment

marshallmain commented Sep 25, 2020

kibanamachine commented Sep 25, 2020

marshallmain commented Sep 25, 2020

elasticmachine commented Sep 23, 2021

		@@ -65,7 +65,7 @@ export const buildRule = ({

		const meta = { ...ruleParams.meta, ...riskScoreMeta, ...severityMeta, ...ruleNameMeta };

		const rule = pickBy<RulesSchema>((value: unknown) => value != null, {

[Detections][EQL] EQL rule execution in detection engine #77419

[Detections][EQL] EQL rule execution in detection engine #77419

Conversation

marshallmain commented Sep 14, 2020 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Sep 22, 2020

andrew-goldstein left a comment

Choose a reason for hiding this comment

marshallmain commented Sep 24, 2020

elasticmachine commented Sep 24, 2020

spong commented Sep 24, 2020 • edited Loading

spong Sep 25, 2020

Choose a reason for hiding this comment

marshallmain Sep 25, 2020 • edited Loading

Choose a reason for hiding this comment

spong Sep 25, 2020

Choose a reason for hiding this comment

spong left a comment

Choose a reason for hiding this comment

marshallmain commented Sep 25, 2020

kibanamachine commented Sep 25, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

page load bundle size

distributable file count

History

marshallmain commented Sep 25, 2020

elasticmachine commented Sep 23, 2021

marshallmain commented Sep 14, 2020 •

edited

Loading

spong commented Sep 24, 2020 •

edited

Loading

marshallmain Sep 25, 2020 •

edited

Loading