Filter non-aggregatable fields from vis editor UI #8421

Bargs · 2016-09-21T23:41:13Z

Fixes #6769 by removing non-aggregatable fields from the vis editor field selector. We get the aggregatable status thanks to elastic/elasticsearch#17980

See commit messages for detailed notes.

Obligatory screenshots:

Notice agent is not aggregatable

It's not available for aggregation! Only the raw version is, as expected

spalger

Looking good, have a few pieces of feedback.

This means that everyone will need to refresh all of their index patterns when they move to 5.0 right? Is there a way we can automate that?

spalger · 2016-09-22T00:16:52Z

src/core_plugins/kibana/server/routes/api/ingest/register_fields.js

+      const callWithRequest = server.plugins.elasticsearch.callWithRequest;
+      const indices = req.params.indices || '';
+
+      callWithRequest(req, 'fieldStats', {


This should probably be returning the promise, in case an error propagates up

spalger · 2016-09-22T00:23:42Z

src/core_plugins/kibana/server/routes/api/ingest/register_fields.js

+
+export function registerFields(server) {
+  server.route({
+    path: '/api/kibana/{indices}/fields',


This feels like a slightly misleading route for what it does.

Yeah, I gave it this name because I plan to expand it to provide all the field info Kibana needs once I remove the mapping cache. That said, I agree it could be misleading atm so I'm open to giving it a different name in the interim. Any good ideas? field_info maybe? field_stats came to mind but seems like a bad idea since I'm filtering out most of the field_stats info.

field_usability_stats? Naming is hard

field_es_compatibility?

field_capabilities?

Bargs · 2016-09-22T19:34:37Z

@spalger I updated the endpoint name and added a mechanism to automatically refresh the field list if aggregatable and searchable are missing. That was a great catch on the issue with upgrading. Should be ready for another look.

spalger · 2016-09-22T20:16:50Z

src/ui/public/index_patterns/_index_pattern.js

-    if (!indexPattern.fields) {
-      return indexPattern.refreshFields();
+
+    let promise = Promise.resolve();


Lets move this to the top of the function so that the return on 104 can also return a promise.

thomasneirynck · 2016-09-22T20:33:39Z

src/core_plugins/kibana/public/management/sections/indices/_indexed_fields.js

@@ -23,8 +23,9 @@ uiModules.get('apps/management')
        { title: 'name' },
        { title: 'type' },
        { title: 'format' },
+        { title: 'searchable' },
+        { title: 'aggregatable' },


I would add a little info blurb here for both too. Something like

'Fields that can be used to aggregate when creating visualizations`

or

`Fields that can be used to search data in the filter bar'

not sure how to word this succinctly.

Apart from that, we probably should add a little blurb about this improvement in the asciidoc too.

thomasneirynck · 2016-09-22T21:26:00Z

LGTM, small suggestion about adding some doc for end-user.

Bargs · 2016-09-22T22:58:33Z

@spalger @thomasneirynck

Tests are passing now
Added helpful tool tips on the index pattern page
Updated that function to always return a promise

Let me know what you think

thomasneirynck · 2016-09-22T23:44:55Z

LGTMagain. Help text is clear and to the point too!

spalger · 2016-09-23T07:05:14Z

👍

This adds a simple API for getting the searchable/aggregatable status of a list of fields in a given index, list of indices, or index pattern. In the future this will probably evolve into a full blown fields info API that we can use when removing the index pattern mapping cache. For now though it's built to provide the minimum info needed to fix elastic#6769 Usage: The API exposes a single GET endpoint. ``` GET /api/kibana/{indices}/field_capabilities ``` `indices` can be a single index, a comma delimited list, or a wildcard pattern Example response: ``` { "fields": { "imsearchable": { "searchable": true, "aggregatable": false }, "imaggregatable": { "searchable": true, "aggregatable": true }, } } ```

Using the field_capabilities API added in the previous commit, this commit enhances the client side index pattern object with information about the searchable and aggregatable status of each field in the index pattern. We then use this information to filter out non-aggregatable fields from the vis editor so that users won't accidentally select them and get nasty errors. An example of a non-aggregatable field would be a `text` field without fielddata enabled (which is the default). I also added the searchable and aggregatable flags to the index pattern page so users can see the status of their fields. I removed the `indexed` column because it was mostly redundant with `searchable` and I needed the horizontal space. The addition of the searchable and aggregatable properties for index pattern fields would require users to manually refresh their field list when upgrading to 5.0. This commit also adds a check for those properties and if they're missing it automatically refreshes the field list for the user in a seamless manner.

--------- **Commit 1:** Add field_capabilities API This adds a simple API for getting the searchable/aggregatable status of a list of fields in a given index, list of indices, or index pattern. In the future this will probably evolve into a full blown fields info API that we can use when removing the index pattern mapping cache. For now though it's built to provide the minimum info needed to fix #6769 Usage: The API exposes a single GET endpoint. ``` GET /api/kibana/{indices}/field_capabilities ``` `indices` can be a single index, a comma delimited list, or a wildcard pattern Example response: ``` { "fields": { "imsearchable": { "searchable": true, "aggregatable": false }, "imaggregatable": { "searchable": true, "aggregatable": true }, } } ``` * Original sha: 1af6b76 * Authored by Matthew Bargar <[email protected]> on 2016-09-21T18:38:34Z **Commit 2:** Filter non-aggregatable fields from vis editor UI Using the field_capabilities API added in the previous commit, this commit enhances the client side index pattern object with information about the searchable and aggregatable status of each field in the index pattern. We then use this information to filter out non-aggregatable fields from the vis editor so that users won't accidentally select them and get nasty errors. An example of a non-aggregatable field would be a `text` field without fielddata enabled (which is the default). I also added the searchable and aggregatable flags to the index pattern page so users can see the status of their fields. I removed the `indexed` column because it was mostly redundant with `searchable` and I needed the horizontal space. The addition of the searchable and aggregatable properties for index pattern fields would require users to manually refresh their field list when upgrading to 5.0. This commit also adds a check for those properties and if they're missing it automatically refreshes the field list for the user in a seamless manner. * Original sha: 4a906f3 * Authored by Matthew Bargar <[email protected]> on 2016-09-21T19:18:10Z

[backport] PR #8421 to 5.0

[backport] PR #8421 to 5.x

LeeDr · 2016-10-07T17:30:29Z

src/core_plugins/kibana/public/visualize/editor/agg_params.js

@@ -138,7 +138,7 @@ uiModules
      }

      function getIndexedFields(param) {
-        let fields = $scope.agg.vis.indexPattern.fields.raw;
+        let fields = _.filter($scope.agg.vis.indexPattern.fields.raw, 'aggregatable');


I'm guessing that this change may have caused scripted fields to no longer show in the list for metric aggregations?
#8585

quite likely...

--------- **Commit 1:** Add field_capabilities API This adds a simple API for getting the searchable/aggregatable status of a list of fields in a given index, list of indices, or index pattern. In the future this will probably evolve into a full blown fields info API that we can use when removing the index pattern mapping cache. For now though it's built to provide the minimum info needed to fix elastic#6769 Usage: The API exposes a single GET endpoint. ``` GET /api/kibana/{indices}/field_capabilities ``` `indices` can be a single index, a comma delimited list, or a wildcard pattern Example response: ``` { "fields": { "imsearchable": { "searchable": true, "aggregatable": false }, "imaggregatable": { "searchable": true, "aggregatable": true }, } } ``` * Original sha: bea909d97634b69f07013485eee41f62d5d017e0 [formerly 1af6b76] * Authored by Matthew Bargar <[email protected]> on 2016-09-21T18:38:34Z **Commit 2:** Filter non-aggregatable fields from vis editor UI Using the field_capabilities API added in the previous commit, this commit enhances the client side index pattern object with information about the searchable and aggregatable status of each field in the index pattern. We then use this information to filter out non-aggregatable fields from the vis editor so that users won't accidentally select them and get nasty errors. An example of a non-aggregatable field would be a `text` field without fielddata enabled (which is the default). I also added the searchable and aggregatable flags to the index pattern page so users can see the status of their fields. I removed the `indexed` column because it was mostly redundant with `searchable` and I needed the horizontal space. The addition of the searchable and aggregatable properties for index pattern fields would require users to manually refresh their field list when upgrading to 5.0. This commit also adds a check for those properties and if they're missing it automatically refreshes the field list for the user in a seamless manner. * Original sha: b823b877f90ce84cb6f789ea90a0fb17e53ad12f [formerly 4a906f3] * Authored by Matthew Bargar <[email protected]> on 2016-09-21T19:18:10Z Former-commit-id: 672f009

[backport] PR elastic#8421 to 5.x Former-commit-id: 481e21b

Bargs added the review label Sep 21, 2016

thomasneirynck self-assigned this Sep 22, 2016

spalger suggested changes Sep 22, 2016

View reviewed changes

Bargs force-pushed the useStats branch from 10c028c to 3514e50 Compare September 22, 2016 15:23

spalger suggested changes Sep 22, 2016

View reviewed changes

thomasneirynck reviewed Sep 22, 2016

View reviewed changes

Bargs force-pushed the useStats branch from 17b51d5 to a788566 Compare September 22, 2016 22:27

spalger approved these changes Sep 23, 2016

View reviewed changes

Bargs added 2 commits September 23, 2016 10:30

Bargs force-pushed the useStats branch from a788566 to 4a906f3 Compare September 23, 2016 14:37

Bargs merged commit a2f5c89 into elastic:master Sep 23, 2016

This was referenced Sep 23, 2016

[backport] PR #8421 to 5.0 #8462

Merged

[backport] PR #8421 to 5.x #8463

Merged

Bargs pushed a commit that referenced this pull request Sep 23, 2016

Merge pull request #8462 from elastic/jasper/backport/8421/5.0

49e858d

[backport] PR #8421 to 5.0

Bargs pushed a commit that referenced this pull request Sep 23, 2016

Merge pull request #8463 from elastic/jasper/backport/8421/5.x

481e21b

[backport] PR #8421 to 5.x

spalger mentioned this pull request Sep 28, 2016

[functionalTests] update column titles after #8421 #8491

Closed

spalger pushed a commit to spalger/kibana that referenced this pull request Sep 28, 2016

[functionalTests] update column titles after elastic#8421

decdf6f

LeeDr reviewed Oct 7, 2016

View reviewed changes

Bargs mentioned this pull request Oct 7, 2016

Flag scripted fields as searchable and aggregatable #8588

Merged

spalger added the v5.0.0-rc1 label Oct 11, 2016

Bargs added the v5.1.0 label Oct 11, 2016

Bargs added the v6.0.0 label Oct 11, 2016

This was referenced Oct 14, 2016

Clicking "Visualize" from Discover field pane results in java.lang.IllegalStateException error #5873

Closed

Conditionally show Visualize button based on aggregatable status #8694

Merged

epixa added the v5.0.0 label Oct 26, 2016

Bargs mentioned this pull request Nov 1, 2016

add top_hit metric #7302

Merged

3 tasks

Bargs mentioned this pull request Nov 14, 2016

Allow to redefine the default index when not found #9028

Closed

epixa added v5.1.1 and removed v5.1.0 labels Dec 8, 2016

airow pushed a commit to airow/kibana that referenced this pull request Feb 16, 2017

Merge pull request elastic#8463 from elastic/jasper/backport/8421/5.x

2c48ab7

[backport] PR elastic#8421 to 5.x Former-commit-id: 481e21b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filter non-aggregatable fields from vis editor UI #8421

Filter non-aggregatable fields from vis editor UI #8421

Bargs commented Sep 21, 2016

spalger left a comment

spalger Sep 22, 2016

spalger Sep 22, 2016

Bargs Sep 22, 2016

spalger Sep 22, 2016

spalger Sep 22, 2016

Bargs Sep 22, 2016

Bargs commented Sep 22, 2016

spalger Sep 22, 2016

thomasneirynck Sep 22, 2016

thomasneirynck commented Sep 22, 2016

Bargs commented Sep 22, 2016

thomasneirynck commented Sep 22, 2016

spalger commented Sep 23, 2016

LeeDr Oct 7, 2016

Bargs Oct 7, 2016

Filter non-aggregatable fields from vis editor UI #8421

Filter non-aggregatable fields from vis editor UI #8421

Conversation

Bargs commented Sep 21, 2016

spalger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bargs commented Sep 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasneirynck commented Sep 22, 2016

Bargs commented Sep 22, 2016

thomasneirynck commented Sep 22, 2016

spalger commented Sep 23, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment