Instrument Kibana with APM

[danielyahn]

Describe the feature:

• High level idea: since Kibana is nodejs app, can it be instrumented by Elastic's APM feature?
• Is Kibana distribution compiled with APM agent?

Describe a specific use case for the feature:

• I'd like to understand user behavior on Kibana.
• For the specific use case, I'm trying to gather insights on queries submitted from Kibana. I successfully created a query audit dashboard by using Elasticsearch's security audit log (https://medium.com/@danielyahn/elasticsearch-query-audit-dashboard-e37fe050b456). But it's missing a crucial piece, which is query performance.

[elasticmachine]

Pinging @elastic/kibana-platform

[watson]

Hi @danielyahn

Thanks for opening this issue. I've been looking into which changes would be required in order to support this use-case, and there's currently a few things we need to add to the Node.js APM Agent in order for this to be properly supported:

• Support for "late configuration" of the agent: Allow config options to be set/changed after agent has started apm-agent-nodejs#877
• Support for instrumenting _msearch queries: [agents] capture request body for more Elasticsearch APIs apm#55

The latter is not strictly needed, but would give a much better experience as a lot of queries in Kibana are performed using _msearch instead of _search.

[zumo64]

Analysing Kibana APM information (queries) would help us troubleshoot a lot of clusters performance issues , identify toxic visualisations and "bad" Kibana users

[watson]

Status update:

We released support for _msearch back in v2.9.0 of the Node.js agent. This just leaves missing support for "late configuration" of the agent: elastic/apm-agent-nodejs#877.

The latter feature is needed to give users a good experience by allowing them to keep the APM configuration inside the Kibana config file. However, if the user is ok with using environment variables to configure the APM agent, then the agent can be used to monitor Kibana already today.

All config options for the APM agent can be set via environment variables. You can see the names of these environment variables listed next to each config option in the documentation. E.g. the most important one is probably serverUrl, which can be set via the environment variable ELASTIC_APM_SERVER_URL.

For a guide on how to install the agent, see the introduction docs:
https://www.elastic.co/guide/en/apm/agent/nodejs/current/intro.html

[lreuven]

@danielyahn @zumo64 did you have a chance to instrument Kibana with the workaround @watson suggested? would you like assistance?

[watson]

I'd like to propose that we add the Node.js agent to the Kibana source code, but in a deactivated state. A user can then activate it using environment variables. This should be an undocumented feature intended for internal debugging and testing, but external users are of course welcome to use it if they know how.

The benefit of adding the agent to the Kibana source code is that it's going to be easier for us internally to use it if we want to. Today, each person who wants to use it would first have to figure out where to install it, then install it etc. If it was already in there, it would hopefully be used much more frequently.

If this is ok with the Kibana team, we can make a PR to the project.

[epixa]

I have no objections to that in theory, so long as late configuration is still the gameplan for getting this to GA. I think Kibana should come orchestrated out of the box for apm, so I don't see any way around shipping the agent along with Kibana itself. It's an oss license, so there isn't a conflict there.

[watson]

Yes, late configuration is still the plan. And I agree we can't GA it before that's possible so that the agent config can be part of the regular Kibana config

[elasticmachine]

Pinging @elastic/apm-ui

[watson]

I've just added this issue to the roadmap of the Node.js APM agent. To scope it, we'll implement an optimistic version that is only configurable via environment variables, as opposed to via the Kibana config file (as described in #32491 (comment))

If we afterward find that we would like to proceed with supporting "late config" which allows configuring the agent via the Kibana config file, we can create another issue for that enhancement specifically.