Realerting is broken when compound query key with array fields is used in elastalert >= 2.19

[ptobis]

Description

An incorrect silence cache key is created, ignoring any query key fields, when the following circumstances occur:

• elastalert2 > 2.18.0
• "realert" is set in the rule,
• compound query key (i.e. created from multiple query key fields) is configured in the rule,
• any query key field refers to an array, that means square brackets are used in at least one of the field names of the compound query key (e.g. "array[0]field")

Example rule

---
name: "test"
type: any
index: "<index>-*"

realert:
    hours: 1

query_key:
    - "some_field"
    - "some_array[0]some_other_field"

filter:
    - query:
        query_string:
          query: "field: value OR otherfield: othervalue"

# Alert configs
alert: "post2"
http_post2_url: "<url>"
http_post2_all_values: true

Result

The silence cache key written to ES contains only the rule name, and completely ignores query key fields. This leads to a situation where all alerts coming from the rule, regardless of query keys, are ignored during the configured time.

Expected result

The silence cache key written to ES contains the rule name and combined values of the compound query key. Only the alerts that match the same query key values are suppressed.

Investigation

The error appears when calling a method get_query_key_value:

elastalert2/elastalert/elastalert.py

Line 908 in 70bc1d8

query_key_value = self.get_query_key_value(rule, match)

because of a new implementation of _find_es_dict_by_key function, that is indirectly used by the get_query_key_value method:

elastalert2/elastalert/util.py

Line 47 in d400d00

def _find_es_dict_by_key(lookup_dict: dict, term: str, string_multi_field_name: str = "keyword") -> tuple[dict, str]:

In case the above circumstances happen, get_query_key_value always return None, that leads to alert suppression not working properly.

The workaround for handling compound query keys that existed in the old version of _find_es_dict_by_key unfortunately has not been implemented in the new function:

elastalert2/elastalert/util.py

Lines 68 to 81 in 3b4608c

	# For compound fieldnames added by ElastAlert.process_hits()
	#
	# For example, when query_key is a list of fieldnames it will insert a term
	# 'key_1,other_fieldname,a_third_name'
	# and if the rule is set for raw_query_keys, the query_key values may end
	# with .keyword it will insert instead something like
	# 'key_1_ip,other_fieldname_number,a_third_name.keyword'
	# and we need to check for that synthentic compound fielname, including the
	# .keyword suffix before contnuing
	#
	# Of course, it also handles happy path, non-ambuiguous fieldnames like
	# 'ip_address' and 'src_displayname' that don't have . or [] characters
	if term in lookup_dict:
	return lookup_dict, term

Please fix the _find_es_dict_by_key function, so it also accepts the compound query key with arrays (square brackets)