Certain Aggregation functions not supported when TSDB datastream is used. #93539
Comments
elasticsearchmachine
added
the
needs:triage
|
Impact Analysis This issue may become significant enough to become as a blocker for TSDB migration of Integration Packages. |
agithomas
changed the title
elasticsearchmachine
added
Team:Analytics
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
|
It appears that some of the aggregation functions are not defined when For example: sum() and average() functions is not defined when In Lens visualisation, there exist no validation to restrict user from selecting aggregation functions that are not defined or supported. This would create error as mentioned in the screenshot below.
It is recommended to limit the selection of such unsupported / undefined aggregate functions in Lens visualisation when @martijnvg , would you be able to help on the supported aggregate functions for
I shall create a Lens Visualisation enhancement in Kibana once your input is available. |
A number of aggregations don't support counter fields, because its computation doesn't make sense on these fields. For example computing an average on a counter doesn't make sense. Relates to elastic#93539
|
Current there are no restrictions on gauge fields. All aggregations support fields with Fields with |
|
I'm going to update the docs to reflect this (^) and make a change to improve the error message. |
|
This error is returned much more often than we had anticipated. The problem is that ES' field capabilities API says the a counter is aggregatable. But in reality ES will only allow a few aggregations to be run on these counter fields. This misinform Lens and many users will run into these unexpected (for users) errors. Whether is makes sense to run aggregations like We will work on a change that will ensure all aggregations will (technically) work on numeric fields with |
|
@martijnvg I'm new to TSDB - can you explain to me why Unfortunately addressing this on the Kibana side looks like it would be a lot of work as it would touch any code that might do aggregations. The code is prepared to handle true or false but nothing more fine grained than that. This also seems like a collaboration issue between ES and Kibana. It would be good to figure out how we got here so we can prevent it in the future. At this point I think our best way forward would be for ES aggregations even when they don't make sense. Its certainly not ideal but we don't have a lot of options for how to move forward. |
|
@mattkime The Because we introduced the
Agreed. I should have reached out.
Yes, and I think just executing the aggregations is the best way forward now. |
No longer offer limited aggregation support counter fields. This extends the counter field support to all aggregation. Closes elastic#93539
…e defined on non tsdb indices. The field caps only reports both attributes on tsdb indices. This change alters the field caps api to also return these attributes when specified on non tsdb indices. Relates to elastic#93539
Fields that have the `time_series_metric` attribute set to `counter` in non tsdb indices should use number value source type instead of counter value source type. Essentially not handling these fields as counters at search time. Relates to elastic#93539
## Summary Part of elastic/elasticsearch#93539 Currently, field caps reports time series counter fields as aggregatable as they support several aggs. Unfortunately, they don't support ALL aggs so its possible create an unsupported agg in Kibana which results in an ES error. In order to prevent this, the data view field list will report all time series counter fields as non-aggregatable until support for specific aggs can be provided throughout Kibana. To test - Create a TSDS with a time series counter field and verify that its not aggregatable in data view management. Also check other apps to make sure the field isn't available for aggregations. https://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-tsds.html ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary Part of elastic/elasticsearch#93539 Currently, field caps reports time series counter fields as aggregatable as they support several aggs. Unfortunately, they don't support ALL aggs so its possible create an unsupported agg in Kibana which results in an ES error. In order to prevent this, the data view field list will report all time series counter fields as non-aggregatable until support for specific aggs can be provided throughout Kibana. To test - Create a TSDS with a time series counter field and verify that its not aggregatable in data view management. Also check other apps to make sure the field isn't available for aggregations. https://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-tsds.html ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 410f019)
…150954) (#151240) # Backport This will backport the following commits from `main` to `8.7`: - [[data views] Time series counter fields are not aggregatable (#150954)](#150954) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Matthew Kime","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-02-15T00:38:23Z","message":"[data views] Time series counter fields are not aggregatable (#150954)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/elasticsearch/issues/93539\r\n\r\nCurrently, field caps reports time series counter fields as aggregatable\r\nas they support several aggs. Unfortunately, they don't support ALL aggs\r\nso its possible create an unsupported agg in Kibana which results in an\r\nES error. In order to prevent this, the data view field list will report\r\nall time series counter fields as non-aggregatable until support for\r\nspecific aggs can be provided throughout Kibana.\r\n\r\nTo test - Create a TSDS with a time series counter field and verify that\r\nits not aggregatable in data view management. Also check other apps to\r\nmake sure the field isn't available for aggregations.\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-tsds.html\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"410f0199408b2c55389ed7d6dfd3170cdede2c84","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Discover","release_note:skip","auto-backport","v8.7.0","v8.8.0","Feature: TSDS"],"number":150954,"url":"https://github.com/elastic/kibana/pull/150954","mergeCommit":{"message":"[data views] Time series counter fields are not aggregatable (#150954)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/elasticsearch/issues/93539\r\n\r\nCurrently, field caps reports time series counter fields as aggregatable\r\nas they support several aggs. Unfortunately, they don't support ALL aggs\r\nso its possible create an unsupported agg in Kibana which results in an\r\nES error. In order to prevent this, the data view field list will report\r\nall time series counter fields as non-aggregatable until support for\r\nspecific aggs can be provided throughout Kibana.\r\n\r\nTo test - Create a TSDS with a time series counter field and verify that\r\nits not aggregatable in data view management. Also check other apps to\r\nmake sure the field isn't available for aggregations.\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-tsds.html\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"410f0199408b2c55389ed7d6dfd3170cdede2c84"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/150954","number":150954,"mergeCommit":{"message":"[data views] Time series counter fields are not aggregatable (#150954)\n\n## Summary\r\n\r\nPart of https://github.com/elastic/elasticsearch/issues/93539\r\n\r\nCurrently, field caps reports time series counter fields as aggregatable\r\nas they support several aggs. Unfortunately, they don't support ALL aggs\r\nso its possible create an unsupported agg in Kibana which results in an\r\nES error. In order to prevent this, the data view field list will report\r\nall time series counter fields as non-aggregatable until support for\r\nspecific aggs can be provided throughout Kibana.\r\n\r\nTo test - Create a TSDS with a time series counter field and verify that\r\nits not aggregatable in data view management. Also check other apps to\r\nmake sure the field isn't available for aggregations.\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-tsds.html\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"410f0199408b2c55389ed7d6dfd3170cdede2c84"}}]}] BACKPORT--> Co-authored-by: Matthew Kime <[email protected]>
Improve field caps api to include what aggregations are supported on a per field bases. For fields that are aggregatable, a list of supported aggregations is included. Closes elastic#93539
Fields that have the time_series_metric attribute set to counter in non tsdb indices should use number value source type instead of counter value source type. Essentially not handling these fields as counters at search time. Relates to #93539
…3800) Fields that have the time_series_metric attribute set to counter in non tsdb indices should use number value source type instead of counter value source type. Essentially not handling these fields as counters at search time. Relates to elastic#93539
|
A small update to this issue. Field defined as counters in non tsdb indices are no longer treated as counters #93800, this should avoid integration issues, because in that case the field caps api doesn't return the I started to work on an change that allows the field caps api to return what aggregations are supported for a specific field: #93884 |
…3800) Fields that have the time_series_metric attribute set to counter in non tsdb indices should use number value source type instead of counter value source type. Essentially not handling these fields as counters at search time. Relates to elastic#93539
…#150954) ## Summary Part of elastic/elasticsearch#93539 Currently, field caps reports time series counter fields as aggregatable as they support several aggs. Unfortunately, they don't support ALL aggs so its possible create an unsupported agg in Kibana which results in an ES error. In order to prevent this, the data view field list will report all time series counter fields as non-aggregatable until support for specific aggs can be provided throughout Kibana. To test - Create a TSDS with a time series counter field and verify that its not aggregatable in data view management. Also check other apps to make sure the field isn't available for aggregations. https://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-tsds.html ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
|
Hi!
Is there a info in what version this will get fixed? |
|
Closing the issue as the fix is available. |
|
I observed the problem using Median function on counter type field, when using 8.7 issue. Reopening the issue Also, please do note that, Median is the default aggregation type that is loaded when the user clicks the Visualize button (appears when the user hover over the field in the Discover view. There are two options here
|
|
I don't know what aggregation the median function is using under the hood. But I don't think supporting median makes sense, for the same reason why |
As is listed here: #93539 (comment) Relates to #93539
As is listed here: elastic#93539 (comment) Relates to elastic#93539
As is listed here: #93539 (comment) Relates to #93539
ElasticSearch Version : elasticsearch:8.7.0-SNAPSHOT (Snapshot downloaded on 6th Feb 2023)
Problem:
The below error occur when using datastream having TSDB enabled.
Imapact
This breaks the Lens visualisation that are having aggregation function on data streams that are TSDB enabled. This is validated using the Oracle Integration TSDB migration activity.
When a problem is seen, visualisation appear as below
Details of error is mentioned below.
Related PR :
When this problem was initially reported, below mentioned PR was created to resolve the problem.
#93311
Initial Impression:
It appears that some of the aggregation functions works with TSDB (Max Value, Last Value) but certain functions do not such as
Mapping Details
Stack Trace
Request
Response
The text was updated successfully, but these errors were encountered: