[Security Solution] [Detections] iterate over default timestamp and optional override in DE loop

[dhurley14]

Summary

Resolves #75382

Partially dependent on #84293 [Merged ✅]

Lots of edge cases to cover in this one so please take your time reviewing.

We now allow for index patterns that match indices with either the optionally provided timestamp override field or the default @timestamp to be used simultaneously when searching for events against our rules. Because some indices in a given index pattern may not have the timestamp override and/or @timestamp field, we now display a "partial failure" status for a rule to show when some indices were missing this and could not be searched against.

For this partial failure to be its most useful, the rule author should have at a minimum view_index_metadata for each index the rule is searching against. To provide more detailed messages it is also suggested for the rule author to have monitor privileges for all indices and monitor for the cluster as well. Without the monitor privilege we cannot provide detailed messages on which specific indices matching the given index patterns are missing the timestamp override field / @timestamp.

Partial failure example setup

Using the following indices

Indices

POST myfakeindex-1/_doc
{
  "message": "hello world 1"
}

POST myfakeindex-2/_doc
{
  "message": "hello world 2",
  "event": {
    "ingested": "2020-12-14T22:31:01.726Z"
  }
}

POST myfakeindex-3/_doc
{
  "message": "hello world 3",
  "@timestamp": "2020-12-14T22:31:01.726Z"
}

and a rule that searched *:* against the index pattern myfa* which matches all three indices above, we are able to search and produce signals against myfakeindex-2 and myfakeindex-3 thus the partial failure where we cannot search against myfakeindex-1 because it is missing both the timestamp override and the default @timestamp field from its mapping.

Screenshot partial failure

---

Full failure example setup

I deleted the indices myfakeindex-2 and myfakeindex-3 so that only myfakeindex-1 was left matching the myfa* index pattern, which did not have the timestamp override field nor the @timestamp field. A situation like this would generate the following error:

Screenshot full failure

---

Full failure no view_index_metadata screenshot

The following is the screenshot a user will see if the rule author did not have view_index_metadata privileges for the indices the rule searched against

Screenshot

Checklist

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[dhurley14]

@elasticmachine merge upstream

[dhurley14]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: ebcff6a

Metrics [docs]

Distributable file count

id	before	after	diff
default	47280	48040	+760

History

• 💚 Build #94810 succeeded 4b97a8115a6f2b9b90871dfb7c57075db708fab6
• 💔 Build #94809 failed 95af27bd3ce3b4a21f49c1805fe8a2f6758c8e3f
• 💚 Build #94482 succeeded 03dee166ddaf390020ca274ef4517384484ce8e2
• 💚 Build #94320 succeeded 1f6733e86d5f3b8da4c274f22f6bb73563c39a07
• 💔 Build #94289 failed 9b75e6483e21546e1c513f735e2622ea65e839ec

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[marshallmain]

Converting the list of index patterns to a list of concrete indices will cause rules to fail on clusters that have rolled indices over enough times, which is likely on large or long running clusters.

The maximum size of 4096 bytes is exceeded with ~130 rollovers of the .siem-signals-default index. With 7 default index patterns, we could only support ~20 concrete indices per pattern before a rule using the default patterns would break, depending on the length of each index name.

I think we should explore options to do the multiple timestamp querying without resorting to expanding the index patterns out into concrete indices. We should be able to build elasticsearch DSL queries that don't break if the index patterns include indices that are missing the expected timestamp field, and then we can build a fallback query that looks for documents that use @timestamp instead of the timestamp override if provided.

[marshallmain]

If any of the index patterns don't have associated indices then the request 404s and prevents the rest of the mappings from being checked. We can solve this by catching the error inside the loop. Probably want to use Promise.all instead of the loop here though, similar to in getIndexesMatchingIndexPatterns

[marshallmain]

The @timestamp key is removed from timestampsAndIndices if all indices have the timestamp override but we still iterate over the whole timestampsToSort array in the top level loop. The result is timestampsAndIndices[timestamp] here can be undefined, which causes Object.keys to throw an error and fail the rule.

[dhurley14]

The issue of expanding too many concrete indices into the query requires a large refactor (#83481 (review)). Closing this PR until I come up with a better solution.