[TSDB][META] Enable supported TSDB counter fields in Kibana

[thomasneirynck]

Describe the feature:

Kibana should not run unsupported aggs on TSDB counter fields.

Context

TSDB counter fields (time_series_type: "counter") do not support all aggregations. When these used, depending on the Kibana app, they can surface errors in unexpected ways.

Two options are currently being proposed:

1) Each app handles counter-fields as "special" case

The default behavior is restored:

• reverts #150954 #152721

Then, each app provides ad-hoc solution when it detects the counter-field.

e.g.:

• ML-datavisualizer: [ML][tsdb] Treat time series counter field as non aggregatable in Index data visualizer #152898
• Maps: maps shoukld offer tsdb counter fields only when its supported by es #152899

2) Elasticsearch publishes supported aggregations

Elasticsearch publishes supported aggs in field_caps. Correspondingly, Kibana would have to change how it uses this new metadata. Depending on the app, this metadata would be treated differently as well:

• DataVisualizer would not run averages by default
• Lens/Maps may hide unsupported aggs as a user-option/have different defaults

A similar solution will be needed for Discover as well. It cannot hide the field. Current state is that it silently swallows an error when opening the field-preview.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[thomasneirynck]

This is blocking #130714 (ie. because counter-types need to be aggregatable)

[nreese]

There is no user facing documentation on what aggregations are supported by which TSDB fields. Either Kibana or Elasticsearch documentation needs to have a table laying out TSDB fields and supported aggregations so users can understand why options are available/unavailable in the UI.

Do you want me to open a separate issue to track? Where should this belong - in Kibana docs or Elasticsearch docs?

[droberts195]

With option 2, Elasticsearch publishes supported aggregations, there could be a CCS complication.

elastic/elasticsearch#93884 (comment) and elastic/elasticsearch#93884 (comment) highlight the potential problem.

If a user is in a CCS environment and is searching a mixture of 8.7 and 8.8 indices then the list of supported aggregations will be empty. This might result in parts of Kibana not permitting any aggregations to be used against the fields. For counter fields this is probably fine - for CCS there's a rule that Kibana doesn't use new Elasticsearch features until 1 minor after they're introduced to avoid accidentally breaking Kibana features when CCS is being used, so it's reasonable that counter fields don't have full functionality in Kibana until the minor after the one where they're introduced. However, what mustn't happen in a mixed 8.7/8.8 CCS environment is that other fields, like simple long and keyword fields, stop working. Field caps would also report an empty list of supported aggregations for these simple fields in a mixed 8.7/8.8 CCS environment. This implies that in 8.8 no part of Kibana should take the list of supported aggregations to be the actual supported aggregations for any field types other than counter. It will cause far more user dissatisfaction if functionality that uses aggregations on simple long or keyword fields stops working in mixed 8.7/8.8 CCS environments than if the brand new feature isn't supported everywhere in Kibana in 8.8.

[droberts195]

There's a suggestion on what to do about the CCS complication in elastic/elasticsearch#93884 (comment). There are different ways to approach it, but the important thing is that there is a compatible approach between Elasticsearch and Kibana.

[thomasneirynck]

Discussed this offline with @ppisljar. I'm +1 on moving with first approach, Each app handles counter-fields as "special" case for 8.8, and potentially moving forward with approach2 at a later date.

cc @martijnvg @droberts195 @jgowdyelastic @stratoula

[davismcphee]

Apologies if I'm missing something obvious here, but does anyone know where to find a list of which aggregations are or aren't supported by counter fields?

[nreese]

Apologies if I'm missing something obvious here, but does anyone know where to find a list of which aggregations are or aren't supported by counter fields?

I was also not able to find this information and neither are our users - #152912 (comment)

[thomasneirynck]

@davismcphee @reese They are listed here elastic/elasticsearch#93539 (comment) . It seems @martijnvg is going to update the docs (elastic/elasticsearch#93539 (comment))

[ThomThomson]

Removing this from the Presentation team project as it seems like our tasks here are completed in #152899.

[timductive]

Closing this issue as TSDB counter fields are now supported in Kibana. Please re-open if I missed any remaining tasks.