[Discuss] Add service.name to service based integrations

[sorenlouv]

The concept of "services" was originally introduced in APM but should not be limited to this domain. Other resources like databases, queues and web servers are also types of services that should be named.
Services in APM are identified using the ECS field service.name, and we can use this for identifying other types of services too.

Service-based integrations like Nginx, Postgresql and MongoDB ingest logs and metrics but the data is not annotated with service.name. There are 3 ways we can apply service.name to integrations:

• Static: "nginx" for the Nginx integration, "postgresql" for the Postgresql integration an so on
• Inferred: The service name can be inferred from standardized/known fields. For example, Kubernetes has recommended labels that "describe applications in a way that can be queried". The service name can therefore be derived from app.kubernetes.io/name if set.
• Configurable: The user can specify the service name they prefer at the time of installing the integration.

Related

• Add service.name with the value of app.kubernetes.io/name label or kubernetes.container.name field, in this order of preference ([Kubernetes] Reroute container logs based on pod annotations #7118)

[ruflin]

Static

I suggest to first focus on the static integrations. AFAIK today the package-spec doesn't have anything around adding a service name, so the only way to add it would be as part of the ingest pipeline. This is not very developer friendly. Instead a package developer should be able to set it either on the package or dataset level. Then Fleet figures out the best way to set it.

For the static use case, I wonder if it could be set as a constant_keyword as it will never change. Is this correct? Will service.name for nginx also be nginx or do users have some config options on this like calling it nginx-prod? What other ECS fields are used here?

Inferred

This sounds more like post processing / magic and not sure if this should be put as a burden to figure out these fields during package development. Could we add a flow around this to Discover? If we find some of these fields in documents, in Discover we ask uses if we should add the service.name accordingly to the dataset. How we do this, has to be figured out.

Configurable

I assume, this applies to integrations that users create in Kibana. Not possible today, but hopefully coming soon. @weltenwort .

[felixbarny]

Will service.name for nginx also be nginx or do users have some config options on this like calling it nginx-prod?

I think it's important that users can customize the service.name to differentiate different nginx services that are used different context.

Inferred: The service name can be inferred from standardized/known fields. For example, Kubernetes has recommended labels that "describe applications in a way that can be queried". The service name can therefore be derived from app.kubernetes.io/name if set.

Related issue: #6845

[akhileshpok]

cc @SubhrataK

[weltenwort]

I assume, this applies to integrations that users create in Kibana. Not possible today, but hopefully coming soon.

Our custom logs on-boarding supports this already:

It currently adds a processor on the edge:

        processors:
          - add_fields:
              target: service
              fields:
                name: test-service-2

[felixbarny]

This is a good start but in order to make the observability experience more service-centric (dare I say asset-centric), I think we should also add service names to all of the integrations that are targeting specific technologies.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!