[Security Solution] [Detections] Multiple timestamp fields #86368

dhurley14 · 2020-12-17T19:49:34Z

Summary

Resolves #75382

If the timestamp override field is missing for a given index pattern, utilize a secondary sort field of @timestamp to perform the search_after and sort on.

This will also set a failure status for rules and log the failure to our rule status SO with the concrete index name and the timestamp field missing from that index. My plan is to get this reviewed then incorporate the partial failure status at a later date as that piece is not mission-critical at the moment.

Testing:

Using the following indices

Indices

POST myfakeindex-1/_doc
{
  "message": "hello world 1"
}

POST myfakeindex-2/_doc
{
  "message": "hello world 2",
  "event": {
    "ingested": "2020-12-14T22:31:01.726Z"
  }
}

POST myfakeindex-3/_doc
{
  "message": "hello world 3",
  "@timestamp": "2020-12-14T22:31:01.726Z"
}

and a rule that searched *:* against the index pattern myfa* which matches all three indices above, we are able to search and produce signals against myfakeindex-2 and myfakeindex-3 thus the partial failure where we cannot search against myfakeindex-1 because it is missing both the timestamp override and the default @timestamp field from its mapping.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

...ck/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.ts

x-pack/test/detection_engine_api_integration/utils.ts

...ck/plugins/security_solution/server/lib/detection_engine/signals/search_after_bulk_create.ts

…l test for this

…in the secondary search. Without updating the total hits field, we could be finding events but not indexing them based on the bulk create logic

…nd excluding documents with timestamp override field

…ak out of loop during secondary search after

…esent

…earch after calls

…t sort ids

kibanamachine · 2020-12-24T00:28:31Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 14f9c77

Metrics [docs]

Distributable file count

id	before	after	diff
`default`	47260	48020	+760

History

💚 Build #96087 succeeded 3c1a9ed7fb5d6ac00b62c44fe66e1df183b530b2
💔 Build #96063 failed bd3031c7093bad8b5a9ee627034d82a6a3b801a6
💚 Build #95904 succeeded a0438485f830e9a0dece0e72e5c8aa1bef56ab80
💚 Build #95883 succeeded f0759b75f3e357e8e93f515d345d2e2385e42add
💚 Build #95662 succeeded e14334a6773976b978103129efbf9f2c8b435c5f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@timestamp

…6368) * query timestamp override and default @timestamp field, adds functional test for this * fix logic for when to filter out timestamp override documents * update the total hits field of the search result if we find hits within the secondary search. Without updating the total hits field, we could be finding events but not indexing them based on the bulk create logic * update integration test, updates logic for performing second search and excluding documents with timestamp override field * cleanup comments, remove commented out console logs, fix logic to break out of loop during secondary search after * default param to 'succeeded' * remove commented out code * always perform a secondary search when timestamp override field is present * perf improvement and fix bug where sortIds were being mixed between search after calls * set sortIds to undefined when not present on search result * exit loop and prevent extraneous searches from occurring if we exhaust sort ids

kibanamachine · 2020-12-28T02:54:39Z