[SIEM] [Detection Engine] Adds filtering abilities to the KQL REST API #49451

FrankHassanabad · 2019-10-27T18:00:01Z

Summary

Removes the older beginner KQL type of signal creation in favor of newer version with filtering
Adds ability to create KQL or lucene queries that will work with the UI filters
UI state with the filters are now savable to re-hydrate UI's on the front end
Adds saved_id ability so the UI can tether dynamic saved queries with signals
Changed it to test as it is not the alias we use for tests
Updated script which converts older saved searches to work with newer mechanism
Fixed script to accept proper ndjson lines
Adds validation unit tests for the endpoint
Increases validation strictness of the endpoints
Adds more data scripts for testing scenarios
[SIEM][Detection Engine] Create REST API endpoint for KQL signals #47013

Testing

Run ./hard_reset.sh script
Test with both algorithms through this toggle before starting kibana:
export USE_REINDEX_API=true
Convert older saved searches to compatible new query filters by running:
./convert_saved_search_to_signals.sh ~/projects/saved_searches /tmp/signals
Post them./post_signal.sh /tmp/signals/*.json
Hard reset again
Test smaller set of signals and REST endpoints using the typical scripts of:

./post_signal.sh
./read_signal.sh
./find_signals.sh
./update_signal.sh
./delete_signal.sh

or test using POSTMAN, etc... If you want to test validation. If you see any validation issues let me know as I have validation testing files and can easily fix them add another unit test to the growing large collection we have now.

Change in your advanced settings of SIEM to use your signals index you configured for verification that the signals show up.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~- [ ] This was checked for cross-browser compatibility, including a check against IE11~~

~~- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~

~~- [ ] Documentation was added for features that require explanation or tutorials~~

Unit or functional tests were updated or added to match the most common scenarios

~~- [ ] This was checked for keyboard-only and screenreader accessibility~~

For maintainers

~~- [ ] This was checked for breaking API changes and was labeled appropriately~~

~~- [ ] This includes a feature addition or change that requires a release note and was labeled appropriately~~

…s initial wip

FrankHassanabad · 2019-10-27T18:05:58Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/get_filter.test.ts

+      ).rejects.toThrow('savedId parameter should be defined');
+    });
+  });
+});


Expand this file if github is trying to hide it as this is all important tests for the system.

nice tests 👍

FrankHassanabad · 2019-10-27T18:06:57Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/schemas.test.ts

+      expect(findSignalsSchema.validate<Partial<FindParamsRest>>({}).value.page).toEqual(1);
+    });
+  });
+});


Expand this file if github is trying to hide it as this is all important tests for the system.

elasticmachine · 2019-10-27T19:17:25Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 17ad889

elasticmachine · 2019-10-28T01:46:54Z

Pinging @elastic/siem (Team:SIEM)

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/schemas.ts

x-pack/legacy/plugins/siem/scripts/convert_saved_search_to_signals.js

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/get_filter.ts

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/create_signals.route.test.ts

dhurley14

Some minor comments but this looks nice and clean!

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/types.ts

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/create_signals_route.ts

stephmilovic

This is my first time reviewing a Detection Engine PR, so I spent a good amount of time this morning digging around and seeing how things work. This PR seems to be an excellent addition. I appreciate the cleanup around tests especially. LGTM!

elasticmachine · 2019-10-28T22:37:16Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d9082d9

elasticmachine · 2019-10-28T23:04:57Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 87b90a3

elastic#49451) ## Summary * Removes the older beginner KQL type of signal creation in favor of newer version with filtering * Adds ability to create KQL or lucene queries that will work with the UI filters * UI state with the filters are now savable to re-hydrate UI's on the front end * Adds `saved_id` ability so the UI can tether dynamic saved queries with signals * Changed `it` to `test` as `it` is not the alias we use for tests * Updated script which converts older saved searches to work with newer mechanism * Fixed script to accept proper ndjson lines * Adds validation unit tests for the endpoint * Increases validation strictness of the endpoints * Adds more data scripts for testing scenarios * elastic#47013 ## Testing * Run `./hard_reset.sh` script * Test with both algorithms through this toggle before starting kibana: `export USE_REINDEX_API=true` * Convert older saved searches to compatible new query filters by running: `./convert_saved_search_to_signals.sh ~/projects/saved_searches /tmp/signals` * Post them`./post_signal.sh /tmp/signals/*.json` * Hard reset again * Test smaller set of signals and REST endpoints using the typical scripts of: ```sh ./post_signal.sh ./read_signal.sh ./find_signals.sh ./update_signal.sh ./delete_signal.sh ``` or test using POSTMAN, etc... If you want to test validation. If you see any validation issues let me know as I have validation testing files and can easily fix them add another unit test to the growing large collection we have now. Change in your advanced settings of SIEM to use your signals index you configured for verification that the signals show up. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

#49451) (#49583) ## Summary * Removes the older beginner KQL type of signal creation in favor of newer version with filtering * Adds ability to create KQL or lucene queries that will work with the UI filters * UI state with the filters are now savable to re-hydrate UI's on the front end * Adds `saved_id` ability so the UI can tether dynamic saved queries with signals * Changed `it` to `test` as `it` is not the alias we use for tests * Updated script which converts older saved searches to work with newer mechanism * Fixed script to accept proper ndjson lines * Adds validation unit tests for the endpoint * Increases validation strictness of the endpoints * Adds more data scripts for testing scenarios * #47013 ## Testing * Run `./hard_reset.sh` script * Test with both algorithms through this toggle before starting kibana: `export USE_REINDEX_API=true` * Convert older saved searches to compatible new query filters by running: `./convert_saved_search_to_signals.sh ~/projects/saved_searches /tmp/signals` * Post them`./post_signal.sh /tmp/signals/*.json` * Hard reset again * Test smaller set of signals and REST endpoints using the typical scripts of: ```sh ./post_signal.sh ./read_signal.sh ./find_signals.sh ./update_signal.sh ./delete_signal.sh ``` or test using POSTMAN, etc... If you want to test validation. If you see any validation issues let me know as I have validation testing files and can easily fix them add another unit test to the growing large collection we have now. Change in your advanced settings of SIEM to use your signals index you configured for verification that the signals show up. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

FrankHassanabad added 14 commits October 23, 2019 17:24

wip (lots of console statements and kind of gross)

ea4e90c

more wip

6c30e51

Merge branch 'master' into update-kql-filtering

0e282fb

Merge branch 'master' into update-kql-filtering

572b9e4

seperated the saved query for a regular query per feedback and this i…

72ce2a3

…s initial wip

Removed console statements

fdcb521

Soldified more of the filters and other formats to use less data types

9399acc

Updated tests and scripts to work with all the new code

ff95c39

More tests and fixes to the schema and REST routes

4d67c11

Fixed savedId to be camel case and not push down underscores

867db3d

More tests added

1124a56

Added getFilter unit tests

e60de98

Merge branch 'master' into update-kql-filtering

8f7ca9a

Turned of alerting for tests to pass

17ad889

FrankHassanabad self-assigned this Oct 27, 2019

FrankHassanabad added v7.6.0 v8.0.0 labels Oct 27, 2019

FrankHassanabad changed the title ~~Update kql filtering~~ [SIEM] [Detection Engine] Adds filtering abilities to KQL searches Oct 27, 2019

FrankHassanabad commented Oct 27, 2019

View reviewed changes

FrankHassanabad added the release_note:enhancement label Oct 27, 2019

FrankHassanabad marked this pull request as ready for review October 27, 2019 18:16

FrankHassanabad changed the title ~~[SIEM] [Detection Engine] Adds filtering abilities to KQL searches~~ [SIEM] [Detection Engine] Adds filtering abilities to the KQL REST API Oct 28, 2019

FrankHassanabad added the Team:SIEM label Oct 28, 2019

dhurley14 reviewed Oct 28, 2019

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/schemas.ts Show resolved Hide resolved

stephmilovic reviewed Oct 28, 2019

View reviewed changes

x-pack/legacy/plugins/siem/scripts/convert_saved_search_to_signals.js Outdated Show resolved Hide resolved

stephmilovic reviewed Oct 28, 2019

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/get_filter.ts Outdated Show resolved Hide resolved

dhurley14 reviewed Oct 28, 2019

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/create_signals.route.test.ts Outdated Show resolved Hide resolved

dhurley14 approved these changes Oct 28, 2019

View reviewed changes

stephmilovic reviewed Oct 28, 2019

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/alerts/types.ts Outdated Show resolved Hide resolved

stephmilovic reviewed Oct 28, 2019

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/create_signals_route.ts Show resolved Hide resolved

stephmilovic approved these changes Oct 28, 2019

View reviewed changes

FrankHassanabad added 4 commits October 28, 2019 15:05

Improvements from the review

57a95b5

Fix accidently pressing random keys there whoops

481dd56

Merge branch 'master' into update-kql-filtering

d9082d9

fixed more payload tests from PR suggestions

87b90a3

FrankHassanabad merged commit a4f37cd into elastic:master Oct 28, 2019

FrankHassanabad deleted the update-kql-filtering branch October 28, 2019 23:27

FrankHassanabad mentioned this pull request Oct 28, 2019

[7.x] [SIEM] [Detection Engine] Adds filtering abilities to the KQL REST API (#49451) #49583

Merged

pmuellr mentioned this pull request Oct 29, 2019

[Task manager] Parallelise task execution #48053

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM] [Detection Engine] Adds filtering abilities to the KQL REST API #49451

[SIEM] [Detection Engine] Adds filtering abilities to the KQL REST API #49451

FrankHassanabad commented Oct 27, 2019 •

edited

Loading

FrankHassanabad Oct 27, 2019

stephmilovic Oct 28, 2019

FrankHassanabad Oct 27, 2019

elasticmachine commented Oct 27, 2019

elasticmachine commented Oct 28, 2019

dhurley14 left a comment

stephmilovic left a comment

elasticmachine commented Oct 28, 2019

elasticmachine commented Oct 28, 2019

[SIEM] [Detection Engine] Adds filtering abilities to the KQL REST API #49451

[SIEM] [Detection Engine] Adds filtering abilities to the KQL REST API #49451

Conversation

FrankHassanabad commented Oct 27, 2019 • edited Loading

Summary

Testing

Checklist

For maintainers

FrankHassanabad Oct 27, 2019

Choose a reason for hiding this comment

stephmilovic Oct 28, 2019

Choose a reason for hiding this comment

FrankHassanabad Oct 27, 2019

Choose a reason for hiding this comment

elasticmachine commented Oct 27, 2019

💚 Build Succeeded

elasticmachine commented Oct 28, 2019

dhurley14 left a comment

Choose a reason for hiding this comment

stephmilovic left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 28, 2019

💚 Build Succeeded

elasticmachine commented Oct 28, 2019

💚 Build Succeeded

FrankHassanabad commented Oct 27, 2019 •

edited

Loading