Adjust ECS dynamic templates to support subobjects: false

[eyalkoren]

We cannot yet set subobjects: false by default at the root level for logs data streams, unless we either modify all shippers to flatten all documents or we automatically flatten objects in Elasticsearch.
However, we can already adjust the ECS dynamic templates to support that, which is what this PR does.

In order to test it, I am using a flatten document.

As I wrote in the test- it is not useable at the moment because setting subobjects: false through custom mappings fails with this error:

        "type" : "illegal_argument_exception",
        "reason" : "updating component template [logs@custom] results in invalid composable template [logs] after templates are merged",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "composable template [logs] template after composition with component templates [logs-mappings, logs-settings, logs@custom, ecs@dynamic_templates] is invalid",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "invalid composite mappings for [logs]",
            "caused_by" : {
              "type" : "mapper_parsing_exception",
              "reason" : "Failed to parse mapping: Tried to add subobject [data_stream] to object [_doc] which does not support subobjects",
              "caused_by" : {
                "type" : "mapper_parsing_exception",
                "reason" : "Tried to add subobject [data_stream] to object [_doc] which does not support subobjects",

@jbaiera if you find out why and how to avoid that- please let me know and I'll adjust what's needed.

In the mean time, this test can be executed by adding subobjects: false to logs-mappings.

@P1llus could you run your exhaustive ECS test with these dynamic templates to see that they are still valid for non-flatted as well?

[elasticsearchmachine]

Hi @eyalkoren, I've created a changelog YAML for you.

[eyalkoren]

Worked around the issue described in #96712 (comment) by creating a dedicated template for the test, so this should be ready for review.
Thanks @felixbarny for the advice 🙏 ❤️

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[felixbarny]

The error you encountered sounds like a bug in how subobjects: false is handled. Please file a bug report for that.

[eyalkoren]

The error you encountered sounds like a bug in how subobjects: false is handled. Please file a bug report for that.

Opened #96768

[jbaiera]

LGTM

[ruflin]

LGTM

I find it slightly surprising that when we would use subobject: false that slightly different templates are required. This is ok for our template as we know the details but what about users. Is there a plan to make templates identical no matter if subobject is used or not?

[eyalkoren]

I find it slightly surprising that when we would use subobject: false that slightly different templates are required. This is ok for our template as we know the details but what about users.

It was somewhat expected that we'd need to adjust, given the way that dynamic templates' match and path_match work.

Is there a plan to make templates identical no matter if subobject is used or not?

There is already only one set of dynamic ECS templates that support both- note that the new 250_logs_no_subobjects.yml test and the former 240_logs_ecs_mappings.yml tests all pass with the same mappings.

One of the things I discussed today with @P1llus is extending his test tool to create a flatten version of all ECS mappings in addition to the nested one it currently creates, thus enabling us test our mappings with subobjects: false even before we make it the default.

[eyalkoren]

I modified @P1llus's test to use the final form of the ECS dynamic templates as we eventually merged them and ran it with the adjusted ECS mappings of this PR to verify that we don't have regression in this regard. Results seem consistent with the adjusted version:

Tested 1584/1581 Fields
Matched fields: 1581
Ignored fields: 3
Missmatched fields: 0

[eyalkoren]

With my latest addition to the test, I was able to verify that when generating a document that contains the entire ECS fields in a flattened form, the mappings proposed in this PR work as expected with subobjects: false.
Therefore, I feel it's safe enough to merge at this point.
@P1llus if you get a chance to take another 👀 , even after merging, that would be great.