[APM] Annotations APIs

[dgieselaar]

Create APIs needed to store and retrieve annotations for all observability plugins.

1. Add an Observability server plugin (we already have a client plugin)
2. Create annotations index on Kibana startup
3. Register APIs to create/read/delete/update annotations
4. Show annotations in APM charts if available (for any timeframe, they will take precedence over user-created deployment annotations).

See below for details.

Annotations

Annotations, or markers, can either be created by the user (user annotations), like a note, or derived from other sources (data annotations), like APM events.

User annotations

User annotations are created and managed by the users of Kibana. A user can create an annotation in all of the observability apps as long as the specific view supports it. The annotation that is created will support both tags and (ECS) metadata (context values), which enables it to be shared across applications and views.

When an annotation is created, context values and tags for the current view should be added to the annotation. Consider a user adding an annotation in the APM transaction duration chart for the opbeans-java - the annotation created would have a @timestamp property (for when it happened), the apm and apm/service/transaction_duration tags, plus { service: { name: 'opbeans-java' } } as context values. These would represent the application, the view / component, and contextual values for the context the annotation was created in.

The presence of context values and tags allows two things: the application for which the annotation was created can recall it; and other applications could surface annotations that are useful in their context. For instance, APM service annotations could be shown in Uptime's monitor view (if we can somehow correlate that data, or the user explicitly specifies it).

Deployment annotations

As said before, deployment annotations are currently derived from APM processor events, by running a min/max aggregation on service.version. This is not desirable for a couple of reasons. First, this requires any observability application that wants to use deployment annotations to have APM events. Second, a min/max aggregation could be slow when processing large data sets. Third, it's very limited, in the sense that there's no way to tell when a deploy is considered finished, to how many instances it has been rolled out, etc. A better solution would be to store deployment events separately, and derive annotations from it.

When indexing a deployment event, it can be tagged as well, in addition to supporting contextual values - similar to user annotations. This would allow our users to freely associate deployment events with Observability UIs. E.g., if they want to associate it with an Uptime monitor, they would add monitor.id; if it's APM, they'd need service.name as a contextual value.

Derived annotations

Annotations could also be derived from other sources. Consider showing fired alerts for an Uptime monitor. In this case, the history of fired alerts could be stored in the event log, and this event log would have to be queried to be able to show annotations based on it. Ideally, any API that would allow us to create and return new annotations, would also be able to surface annotations from various sources.

Storing annotations

Annotations are stored as ES documents in an annotations index, in accordance with ECS where possible. This allows data to be easily linked together. Annotations can be queried like any other ES document, including KQL.

There are two ways to create an annotation: a Kibana API, and by simply indexing documents into Elasticsearch. The Kibana API provides additional validation and pre-filling of certain values.

Schema

Annotation documents use ECS where possible. Additionally, metadata about the annotation is placed under the annotation key. Here's a sample document for a deployment annotation, where (@timestamp is when the event happened, and event.created signals the time it was stored):

{
  "annotation": {
    "type": "deployment"
  },
  "@timestamp": "2020-01-29T10:57:03.902Z",
  "event": {
    "created": "2020-01-29T10:57:12.113Z"
  },
  "service": {
    "name": "opbeans-java",
    "version": "1.0.0"
  },
  "monitor": {
    "id": "opbeans-java"
  }
}

And a user annotation:

{
  "annotation": {
    "type": "user",
    "content": "You Know, for Search",
    "tags": [ "apm", "apm/service/transaction_duration" ]
  },
  "@timestamp": "2020-01-29T10:57:03.902Z",
  "event": {
    "created": "2020-01-29T10:57:12.113Z"
  },
  "service": {
    "name": "opbeans-java",
    "version": "1.0.0"
  },
  "monitor": {
    "id": "opbeans-java"
  },
  "user": {
    "username": "kimchy"
  }
}

The Kibana API will roughly accept the same document, except it will add a timestamp, and some metadata about the current user if it's a user annotation.

Index management

On startup, the Kibana server will create an annotations index and template if it does not exist yet. The name of the index should be configurable, with a sane default (like observability-annotations).

Annotations plugin

We can leverage the existing Observability plugin to create the annotations index, and register the Kibana API. This plugin should also receive the config setting for the annotation index (observability.annotationsIndex).

Querying annotations

Annotations can be queried by either calling Elasticsearch directly, or using the Observability Annotations plugin (either via the Kibana API or by a server module). Querying annotations would happen by providing a time range, types (annotations are stored annotations), and a query (KQL), all optional:

GET http://localhost:5601/api/observability/annotations
{
  from: '2020-01-29T10:57:03.902Z',
  to: ''2020-01-30T10:57:03.902Z'',
  types: [ 'annotations', 'alerts' ],
  query: 'service.name:"opbeans-java" AND annotation.tags:"apm" AND annotation.tags:"apm/service/transaction_duration"'
}

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)