EQL: null handling in ES EQL

[astefan]

EQL treats a non-existent value (that can be associated with a null to some degree) in a "lenient" way. For example, length will return 0 if there is an empty string or non-existent string. Should length return null instead of 0?

A more interesting scenario is the one of functions that return a boolean value (startsWith is one of them) that receive a null parameter? Should they return null or a true/false value?

From #53999:

null behavior:

wildcard(missing/null, ...) ==> null
wildcard(string, ...) => true or false
wildcard(non-null, non-string) => null (if necessary)
this shouldn't be possible when compiling to match and using an indexed field. but can we query non-indexed fields? would a document ever have a non-string value?

The above suggestion does bring forth another scenario: functions dealing with data types they are not supposed to deal with - should the function return null or a "lenient" value (like 0 in the case of length(integer) for example) or throw an exception?

I created this issue to open discussion and reach a conclusion about "null" handling (both as parameters and as a return value) in ES EQL:

• functions dealing with null parameters should return null or a default value depending on the function?
• functions dealing with parameter types other than the ones supported by those functions should be "silent" about this incompatibility and return a default value or throw an exception?
• the decision to return null should probably be accompanied by the introduction of a new function, that should check for a null value. Otherwise, the null output of functions will not be useful in queries

[elasticmachine]

Pinging @elastic/es-ql (:Query Languages/EQL)

[astefan]

A valid scenario that needs a solution: file where opcode=0 and indexOf(file_name, 'plore') == 2 and not indexOf(file_name, '.pf')
In this case not indexOf should return a boolean value. But indexOf is in fact a function that has an output of type number. In reality, not indexOf is a variant of indexOf IS NOT NULL.

[astefan]

Another interesting scenario - the string function - where, if the object to be represented as a string is null, the "null" string is returned, instead of null object.

Update: this is no longer the case, similar with concat

[astefan]

• null comparison (is null / is not null) is implicit in EQL and it works at the moment.
• allow null as a valid parameter in functions (even in those that return true/false and consider null as false)
• should pursue null equality, which should be already supported as part of equals.
• concat is a special case where a null is treated as empty string

[rw-access]

I think we can close this with #54544, since all of the null handling should be consistent now.