Add messaging fields to schema

[eyalkoren]

Description of the issue

Adding dedicated fields to schema to support messaging context.

What we are voting on

New types

Background

Message send/publish events can only be captured as spans if occurring within a traced transaction.
Message consumption can be divided into two types: passive, where you would implement a listener that is called once a message is available, and active, where the queue is being polled (blocking or non-blocking).
Passive consumption typically results in a Messaging transaction and is pretty straightforward to trace.
Message polling can be done within a traced transaction, in which case it will result in a messaging span, or it can be the initiating event for a message handling flow. Capturing polling spans is also mostly straightforward. For polling-based transactions, our goal is to capture the message handling flow, which typically starts after the polling action exits, returning a message. This may be tricky if the handling flow is not implemented within a well defined API. The actual implementation for that may be different for different clients. Whenever it is not possible (or too risky), we may consider capturing the message polling action itself, but this should be decided for each case separately and should only be used as a last resort. In any case, we should not create a transaction based on polling APIs if the polling action did not result with a message (as opposed to polling spans, where we want to capture such as well).

Proposed types

• Transactions: messaging for transaction.type
• Spans: messaging for span.type (both sends and receives-within-a-transaction), the name of the framework for span.subtype (eg jms or kafka) and the operation for span.action (eg send, receive, poll).

New fields

• context.message.queue.name: optional for messaging spans and transactions. Indexed as keyword. Wherever the broker terminology uses "topic", this field will contain the topic name.
• context.message.body: similar to HTTP requests' context.request.body- only fill in messaging-related transactions (ie incoming messages creating a transaction) and not for outgoing messaging spans.
  • Rely on the existing ELASTIC_APM_CAPTURE_BODY agent config option.
  • Only capture UTF-8 encoded message bodies
  • Limit size to 10000 characters. If longer than this size, trim to 9999 and append with ellipsis
  • Storage: same Elasticsearch mapping as context.request.body
• context.message.headers: similar to HTTP requests' context.request.headers- only fill in messaging-related transactions.
  • Rely on the existing ELASTIC_APM_CAPTURE_HEADERS agent config option.
  • Sanitize headers with keys configured through ELASTIC_APM_SANITIZE_FIELD_NAMES
  • Intake: key-value pairs, same like context.request.headers
  • Storage: same mapping as used for context.request.headers.
• context.message.age.ms: a numeric field indicating the message's age in milliseconds. Relevant for transactions and receive spans that receive valid messages. There is no accurate definition as to how this is calculated. If the messaging framework provides a timestamp for the message- agents may use it. Otherwise, the sending agent can add a timestamp indicated as milliseconds since epoch UTC to the message's metadata to be retrieved by the receiving agent. If a timestamp is not available- agents should omit this field. Clock skews between agents are ignored, unless the calculated age (receive-timestamp minus send-timestamp) is negative, in which case the agent should report 0 for this field.

Vote

@elastic/apm-agent-devs and @elastic/apm-server: tick the N/A box if irrelevant for you. Otherwise, link to the relevant issue or PR this and tick "Yes".

Component	Yes	N/A	Link to component issue
.NET
Go
Java			elastic/apm-agent-java#664
Node.js
Python
Ruby
RUM
Server			elastic/apm-server#2697

[simitt]

Storage: same mapping as used for context.request.headers.
Storage: same Elasticsearch mapping as context.request.body

APM Server maps context.request.headers to http.request.headers and context.request.body to http.request.body. I don't think this would be appropriate, we rather have to find some other terminology like messaging and maybe define and populate an additional messaging.type: kafka | amqp | jms | ..

However, this is another discussion that we need to have APM Server side, and maybe even think about adding to ECS, just wanted to point out that ES mapping is not that clear IMO.

[eyalkoren]

@simitt yes, sorry about that, where to locate these fields is not something I can assist with, what I meant to say is that these fields will be used similarly to the HTTP ones, so the type mapping should be similar. Please feel free to update the storage definition to whatever you decide.

Regarding ECS- we discussed that and decided to start without adding it for now. If you think we should, let's discuss further.

[simitt]

context.message.queue.name and context.message.topic.name: required for messaging spans and transactions.

@eyalkoren will the information be sent for transaction and span events? You are comparing it to context.request, which is only sent for transaction events.

[simitt]

@eyalkoren do you think it would be possible and valuable to detect the type of the messaging system? (kafka, amqp, etc)

[eyalkoren]

will the information be sent for transaction and span events? You are comparing it to context.request, which is only sent for transaction events.

I am not comparing context.message.queue.name and context.message.topic.name to context.request, but the other two, that should indeed be sent only for transactions. You are right- context.message.queue.name and context.message.topic.name should be sent both for spans and transactions (in spans, context.message would correspond context.http in this regard). Do you see a problem with that?

do you think it would be possible and valuable to detect the type of the messaging system? (kafka, amqp, etc)

It certainly would make sense to have new types, not only for spans, but for transactions as well. I didn't include it in this issue because we still need to finalize what we do with the transaction types (we did not get to a final conclusion there - see the discussion issue) and because it doesn't require schema changes.
But we did agree to use messaging for span.type (sends and receives within a transaction), the name of the framework for span.subtype (eg jms or kafka) and the operation for span.action eg (send or receive).

[simitt]

I am not comparing context.message.queue.name and context.message.topic.name to context.request, but the other two, that should indeed be sent only for transactions. You are right- context.message.queue.name and context.message.topic.name should be sent both for spans and transactions (in spans, context.message would correspond context.http in this regard). Do you see a problem with that?

No problem, just needed clarification for the apm-server task.

[eyalkoren]

Added a New types section to the proposal.

[eyalkoren]

@elastic/apm-server I started implementing for the Java agent. Is this planned for 7.6?
@formgeist Please take a look - there are new fields to add for Messaging spans and transactions. This is a first step of providing some inherent Messaging support.

[simitt]

@eyalkoren yes it is planned for 7.6, but implementation hasn't started (see elastic/apm-server#2697).

[formgeist]

@eyalkoren Thanks for the heads up - I've created an accompanying Kibana issue to add the fields to the metadata. elastic/kibana#49465

[eyalkoren]

@simitt @formgeist since I am the only one that implemented this so far, I added the context.message.age.ms field to this issue's description, as per the latest requirement. Please review and let me know if a clarification/modification is required.

Since the value "0" is valid and given that this is never an accurate measurement, consider some special handling for 0, for example show ~0. This field is mostly to indicate traces where messages were delayed in a queue for a long time, so the values 0 and 100 (ms) are pretty much the same thing.

[formgeist]

@eyalkoren Thanks for the heads up - I've updated the description of the Kibana issue

[eyalkoren]

@simitt @elastic/apm-ui please note that following @roncohen's request we removed context.message.topic.name, so that now only context.message.queue.name can be sent.

[eyalkoren]

Closing this as we have it all in the spec now.