docs: Finish Application logging docs

[bmorelli25]

Summary

This PR is a follow-up to #2452. Its main purpose is to update the Filebeat configuration with additional details and updated configuration options.

[github-actions]

A documentation preview will be available soon:

• 📚 HTML diff
• 📘 Fleet and Elastic Agent Guide
• 📙 Observability Guide

[bmorelli25]

Removed keys_under_root as new keys go under root by default. This can be changed with target (source), but I didn't think it was necessary to list it here as we're not suggesting changing the default.

[SylvainJuge]

@bmorelli25 do you know which version introduced this change and does it applies to other keys that are injected into the document ?
In the ready-to-use examples that I currently have (initially built for 8.5.3, but now upgraded to 8.6.0):

• fields_under_root: https://github.com/elastic/apm-contrib/blob/b99ee6225f01bad97901396d63a2f29d31a57f2a/apm-agent-java/log-ingest/01-filebeat.yml#L20 (for plain-text logs)
• keys_under_root https://github.com/elastic/apm-contrib/blob/b99ee6225f01bad97901396d63a2f29d31a57f2a/apm-agent-java/log-ingest/02-filebeat.yml#L13 (for nd-json logs)

[bmorelli25]

For nd_json logs, keys_under_root doesn't do anything and never has; a documentation error in the Filebeat docs made it appear necessary. In 8.2, the documentation was updated to show target instead of keys_under_root. If target is not set, fields go to the root of the event by default (source).

For plain text logs, you are correct. fields_under_root is needed to store custom fields as top-level fields in the output document instead of being grouped under a fields sub-dictionary (source).

[SylvainJuge]

In order to have correlation between log documents and APM services, we need to (at least) set the value of service.name (and maybe few other fields but I don't know yet which ones).
This is required for the following strategies:

• plain-text with filebeat
• ECS logging with filebeat:
  • service.name can be set in ECS logging configuration
  • when not set at ECS logging configuration level and deployed with the Java agent, then the Java agent will set service.name with the value from its own configuration.

When using the other strategies (ECS reformatting and Log sending), then the ECS logging library in the agent is configured with the service.name from the agent configuration, hence making it implicit.

Without this, the end-user will only be able to see logs documents in the Logs Stream UI, not in APM.

[SylvainJuge]

The service.name field is currently the main way to correlate the logs, which is properly described in elastic/kibana#148788 description:

Logs annotated with a matching service.name are displayed . Log lines without service.name will be matched using container.id and then host.name (in that order).

I don't think that we need to document the container.id and host.name fallback as the container.id is very unlikely to be set by the end user, but it can be implicitly provided when using the APM agents.

[dedemorton]

Changes LGTM from an editorial perspective.

[bmorelli25]

Thanks!

For plaintext, I added a step on log correlation. For ECS logging with filebeat, it looks like all of the ECS guides mention log correlation, so I'm not sure we need to add it to these docs.

[SylvainJuge]

For the plain-text logs, I think that adding the service.name extra field configuration would be relevant here :

observability-docs/docs/en/observability/tab-widgets/filebeat-logs/content.asciidoc

Line 43 in ce3ebd1

paths: /path/to/logs.log <2>

While there is a mention of service.name in the "Application logs" section, this is quite far away and for plaintext it will always be required (unlike ECS logging where it can be set either through logger configuration or directly by the APM agent), hence usually not required in the filebeat configuration.