Make sure that main ECS fields are mapped in the Elastic Agent integration

[rdner]

Recently we had a report that log.file.path is not mapped in events created in the Elastic-Agent integration (dataset: elastic_agent.endpoint_security-external).

Another report that error.message is not mapped in logs-elastic_agent.filebeat-syslog either.

All the logs-*-* data streams including logs-elastic_agent.endpoint_security-external are created from this logs-template.json

which is composed of these component templates:

• "logs-mappings"
• "logs-settings"
• "logs@custom" (ignored when missing)
• "ecs@dynamic_templates"

The last 2 were added ~4 months ago in this PR (should have been released with 8.9.x) elastic/elasticsearch#96171

Which means the log.file.path field should be mapped via this dynamic mapping:

https://github.com/elastic/elasticsearch/blob/48fcb2e7d4651b95a8e22612e2ec001ad4c4bdc4/x-pack/plugin/core/template-resources/src/main/resources/ecs-dynamic-mappings.json#L100

However, I see that dynamic mappings are disabled in the integration:

integrations/packages/elastic_agent/data_stream/endpoint_sercurity_logs/manifest.yml

Line 7 in ffe9c35

dynamic: false

It seems we have two options (both need to be tested):

1. Set dynamic to true here

   integrations/packages/elastic_agent/data_stream/endpoint_sercurity_logs/manifest.yml

   Line 7 in ffe9c35

   dynamic: false

2. Add a limited list of most important fields (e.g. log.file.path) here https://github.com/elastic/integrations/blob/ffe9c35004696815c24fdb89aa86778d2ea67885/packages/elastic_agent/data_stream/endpoint_sercurity_logs/fields/ecs.yml

(1) might have big impact and lead to performance issues, see #1298 (comment)

(2) is fixing the currently reported symptom instead of the root cause and we might receive more reports in the future.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

I vote for option 2 unless we believe option 1 is totally risk free. We don't seem to get a lot of requests for new fields, so I hesitate to do something that may cause performance problems whenever the elastic_agent integration is used (which is a lot).

[rdner]

We've got another report that error.message is not mapped either.

Mapped only in the fleet server logs

integrations/packages/elastic_agent/data_stream/fleet_server_logs/fields/ecs.yml

Line 17 in 399ac66

- name: error.message

[tammytorbert]

Another field not mapped - host.os.type. Mapping appears to be missing from component template - logs-elastic_agent.filebeat@package