Add ingest timestamp to pipeline

[dgieselaar]

To be able to use transforms (and other post-processing pipelines), an ingest timestamp could be added to the events on ingestion:

https://www.elastic.co/guide/en/elasticsearch/reference/7.4/accessing-data-in-pipelines.html#accessing-ingest-metadata

This allows jobs to collect new documents, either for alerts, transforms or other projections of processor events. The current @timestamp references the time when the event was recorded, not when it was ingested, and is not a reliable way to sort when trying to collect new documents.

As APM Server currently adds the pipeline and the index template, it should probably handle the configuration of the ingest timestamp as well.

We should figure out how to deal with backwards compatibility. If the user is on Kibana 7.6, but on APM Server 7.1, is it possible for Kibana to add both the processor and update the mapping for the APM indices? If we add { ingest: { timestamp: { ... } } } to the mapping from Kibana, what happens when APM server overrides the template again?

cc @elastic/apm-ui

[sorenlouv]

what happens when APM server overrides the template again

APM Server doesn't overwrite the template. But if the user is on 7.4 and upgrades to 7.5 APM Server will create a new template that doesn't have the ingest mapping.

is it possible for Kibana to add both the processor and update the mapping for the APM indices

I think we should be able to do this when Kibana start up. Which would also solve the problem above (we'd require a kibana restart when APM Server is upgraded).

[graphaelli]

elastic/ecs#582 introduces an event.ingested for this purpose. We can add an ingest pipeline (like elastic/beats#14001) to write this field - do we need it for every document or just transactions and errors?

[dgieselaar]

@graphaelli I think just transactions and errors is fine for now. I don't expect us to do any post-processing for spans and metrics any time soon (famous last words).

[dgieselaar]

Awesome! Op wo 5 feb. 2020 19:11 schreef Gil Raphaelli <[email protected]>:

…

Closed #2934 <#2934> via #3279 <#3279>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2934?email_source=notifications&email_token=AACWDXAEOO356TMV3E4JNNDRBL6MLA5CNFSM4JPRK342YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWN4RGRI#event-3011056453>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACWDXB6KTIMB2YBMWJADNTRBL6MLANCNFSM4JPRK34Q> .