[APM] Investigate possibility of using edge metrics to build service map

[dgieselaar]

We currently use an approach where we sample spans based on their fingerprint, via a composite aggregation. In some cases this can be rather slow. We should look into whether we can use service map edge metrics to generate a service map, under the assumption that it will be faster.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[dgieselaar]

These spans record almost the same information as the spans that we use to build the service map. The following fields are available:

• service.name
• span.destination.service.resource
• span.type
• span.subtype

However, we also need trace.id, to be able to download the traces in the second request. @elastic/apm-server is this something that we can add to the span metrics? It would help increase performance of the first query, but it doesn't fundamentally change the approach we have (get trace ids, use a scripted metric aggregation to extract connections from the traces).

[jalvz]

Certainly, feel free to open an issue on apm-server 👍

[dgieselaar]

elastic/apm-server#4371

[axw]

However, we also need trace.id, to be able to download the traces in the second request

What's missing from the span metrics that you extract from trace events? If possible I think we should consider moving away from walking traces, and instead store enough information in the metrics to reconstruct the aggregated graph structure.

[dgieselaar]

@axw Without a trace ID we can't get paths, only connections (one hop downstream). Agree that storing the path (somewhere) is a better approach, as this is essentially just a performance optimisation of an untenable strategy (walking traces).

[dgieselaar]

As it stands, span (service map edge) metrics are not sufficient to build a service map. Discussion continues in elastic/apm#364.