field_capabilites handler swallows an upgrade-related exception from ES

[tsullivan]

Kibana version:
5.3.1

Elasticsearch version:
5.3.1

Browser/Server OS version:
Darwin

Browser version:
Chrome

Original install method (e.g. download page, yum, from source, etc.):
.tar.gz from download

Description of the problem including expected versus actual behavior:
When there is a problem in the Elasticsearch mapping

Steps to reproduce:

1. Starting with ES 2.4.4
2. Create an index mapping with a timestamp, analyzed string field + a geo_point:

PUT index
{"mappings":{"type":{"properties":{"geo_field":{"type":"geo_point"},"text_field":{"type":"string"},"@timestamp":{"type":"date"}}}}}

3. Add a sample document

PUT index/type/1
{"geo_field":"33.8957, -112.0577","text_field":"Hello World!","@timestamp":"2017-04-21T12:37:11.000Z"}

4. Create an index pattern in Kibana for the index
5. Upgrade to 5.3.1 (you can just copy the data directory over)
6. This will trigger _field_stats call on geo_point field broken after upgrading from 2.4.4 -> 5.3.1 elasticsearch#24275
7. Kibana will show confusing error messages, and will swallow the actual field_stats error that's described in _field_stats call on geo_point field broken after upgrading from 2.4.4 -> 5.3.1 elasticsearch#24275

I added console.log logging in the code and found this is the callWithRequest response that Kibana gets at line https://github.com/elastic/kibana/blob/5.3/src/core_plugins/kibana/server/routes/api/ingest/register_field_capabilities.js#L20:

{"_shards":{"total":5,"successful":4,"failed":1,"failures":[{"shard":3,"index":"index","status":"INTERNAL_SERVER_ERROR","reason":{"type":"exception","reason":"java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 5","caused_by":{"type":"execution_exception","reason":"java.lang.ArrayIndexOutOfBoundsException: 5","caused_by":{"type":"array_index_out_of_bounds_exception","reason":"5"}}}}]},"indices":{"_all":{"fields":{}}}}

Errors in browser console (if relevant):

• No errors in the JS console or any HTTP errors
• Discover shows unhelpful errors:
  
• Refreshing the Index Pattern doesn't solve the problem and seems to show the field mappings correctly (which is very strange considering field_capabilities returns nothing:
  

Provide logs and/or server output (if relevant):
With verbose logging, no errors are logged either when trying to load Discover, or when refreshing the Index Pattern

[tsullivan]

Another thing that's odd is my geo_field is checked as being searchable and aggregatable, even after refreshing the index pattern, given that field_capabilities returned nothing:

[Bargs]

Related to #9571 (which is why you're seeing weird errors in Discover). At least one user near the end of that comment thread seems to be running into this issue.

It looks like you're still seeing the geo_field marked as searchable and aggregatable because it's hardcoded to workaround elastic/elasticsearch#20707.

@tsullivan does this only happen when upgrading from 2.4.4 to 5.3.1?

In the short term we should definitely stop swallowing the error, long term it won't be a problem when we switch to the field_caps API.

[Bargs]

So we already handle most errors, but this is a shard failure so it doesn't fail the entire request. I'm wondering if we already have a common way of handling this. @spalger does the ES client handle shard failures at all? Or are we best off extracting them in the success handler?

[Bargs]

As a workaround users may find this community member's script useful to temporarily mark all fields as searchable and aggregatable.

[spalger]

@spalger does the ES client handle shard failures at all?

The es client doesn't do anything special with shard failures. The courier automatically warns when any number of shards fails, but that's obviously not applicable to all requests