[Security Solution] [Detections] Combine multiple timestamp searches into single request #95723

dhurley14 · 2021-03-29T21:59:09Z

Summary

Related issues: elastic/elasticsearch-specification#267

Background

Initial implementation of the ability for rules to query against documents using multiple timestamps resulted in two single search afters being sent in series.

Research

There was some investigation into combining these sort fields into one query so that we do not need to make two separate search requests. The problem with this combination is we need to query documents that use the timestamp override field and simultaneously query documents that do not have the timestamp override field.

To execute this same query in a single request we would need the ability to execute the query using two sort fields where the query has an XOR (query with timestamp override field XOR query with @timestamp field and documents do not have the timestamp override field.)

Proposed solution

Looking into this I feel that the better option (and maybe only option since I have not found a way in Elasticsearch DSL to execute an XOR filter with sorting) is to utilize the msearch api. This supports search_after and allows us to send one single request while keeping the searches logically separated.

The msearch api could benefit rules that require executing the same query with different params such as threat indicator rules but this is out of scope for this PR.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

kibanamachine · 2021-03-29T23:31:42Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 92fbdea
Storybooks Preview
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

Kibana Pipeline / jest / Jest Tests.x-pack/plugins/security_solution/server/lib/detection_engine/signals.create_signals it builds a now-5m up to today filter

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toEqual(expected) // deep equality

- Expected  - 0
+ Received  + 6

@@ -14,21 +14,27 @@
        },
      ],
      "query": Object {
        "bool": Object {
          "filter": Array [
+           Object {
+             "bool": Object {
+               "should": Array [
                  Object {},
                  Object {
                    "bool": Object {
                      "filter": Array [
                        Object {
                          "range": Object {
                            "@timestamp": Object {
                              "format": "strict_date_optional_time",
                              "gte": "now-5m",
                              "lte": "today",
+                           },
                          },
+                       },
+                     ],
                    },
                  },
                ],
              },
            },
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_events_query.test.ts:22:19)
    at Promise.then.completed (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:276:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:216:10)
    at _callCircusTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:212:40)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at _runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:472:34)
    at Object.worker (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/testWorker.js:133:12)

Kibana Pipeline / jest / Jest Tests.x-pack/plugins/security_solution/server/lib/detection_engine/signals.create_signals if searchAfterSortId is an empty string it should not be included

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toEqual(expected) // deep equality

- Expected  - 0
+ Received  + 6

@@ -14,21 +14,27 @@
        },
      ],
      "query": Object {
        "bool": Object {
          "filter": Array [
+           Object {
+             "bool": Object {
+               "should": Array [
                  Object {},
                  Object {
                    "bool": Object {
                      "filter": Array [
                        Object {
                          "range": Object {
                            "@timestamp": Object {
                              "format": "strict_date_optional_time",
                              "gte": "now-5m",
                              "lte": "today",
+                           },
                          },
+                       },
+                     ],
                    },
                  },
                ],
              },
            },
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_events_query.test.ts:87:19)
    at Promise.then.completed (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:276:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:216:10)
    at _callCircusTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:212:40)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at _runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:472:34)
    at Object.worker (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/testWorker.js:133:12)

Kibana Pipeline / jest / Jest Tests.x-pack/plugins/security_solution/server/lib/detection_engine/signals.create_signals if searchAfterSortId is a valid sortId string

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toEqual(expected) // deep equality

- Expected  - 2
+ Received  + 6

@@ -14,21 +14,27 @@
        },
      ],
      "query": Object {
        "bool": Object {
          "filter": Array [
+           Object {
+             "bool": Object {
+               "should": Array [
                  Object {},
                  Object {
                    "bool": Object {
                      "filter": Array [
                        Object {
                          "range": Object {
                            "@timestamp": Object {
                              "format": "strict_date_optional_time",
                              "gte": "now-5m",
                              "lte": "today",
+                           },
+                         },
                        },
+                     ],
                    },
                  },
                ],
              },
            },
@@ -37,13 +43,11 @@
            },
          ],
        },
      },
      "search_after": Array [
-       Array [
        123456789012,
-       ],
      ],
      "sort": Array [
        Object {
          "@timestamp": Object {
            "order": "asc",
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_events_query.test.ts:153:19)
    at Promise.then.completed (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:276:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:216:10)
    at _callCircusTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:212:40)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at _runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:472:34)
    at Object.worker (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/testWorker.js:133:12)

and 8 more failures, only showing the first 3.

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

dhurley14 · 2021-04-01T18:48:57Z

implementing without msearch here: #96078

dhurley14 added 2 commits March 29, 2021 14:10

WIP

62c7a90

updates with changes from detection engine refactor

92fbdea

dhurley14 closed this Apr 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] [Detections] Combine multiple timestamp searches into single request #95723

[Security Solution] [Detections] Combine multiple timestamp searches into single request #95723

dhurley14 commented Mar 29, 2021 •

edited

Loading

kibanamachine commented Mar 29, 2021 •

edited

Loading

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

dhurley14 commented Apr 1, 2021

[Security Solution] [Detections] Combine multiple timestamp searches into single request #95723

[Security Solution] [Detections] Combine multiple timestamp searches into single request #95723

Conversation

dhurley14 commented Mar 29, 2021 • edited Loading

Summary

Background

Research

Proposed solution

Checklist

For maintainers

kibanamachine commented Mar 29, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

Metrics [docs]

dhurley14 commented Apr 1, 2021

dhurley14 commented Mar 29, 2021 •

edited

Loading

kibanamachine commented Mar 29, 2021 •

edited

Loading