Bug: Cannot read property 'result' of undefined

[gmoskovicz]

Kibana version: 6.1.1

Elasticsearch version: 6.1.1

Server OS version: Any

Browser version: Any

Browser OS version: Any

Original install method (e.g. download page, yum, from source, etc.): Zip

Description of the problem including expected versus actual behavior:

Kibana throws an error when it doesn't seems to be able to retrieve the results from the kibana index. This seems to happen when all shard fails for a 429:

2018-02-26T21:27:59.229296300Z {"type":"error","@timestamp":"2018-02-26T21:27:59Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":19,"level":"error","error":{"message":"all shards failed: [search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"size\":0,\"from\":0},\"body\":\"{\\\"version\\\":true,\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":[{\\\"term\\\":{\\\"type\\\":\\\"dashboard\\\"}}]}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n    at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n    at emitNone (events.js:91:20)\n    at IncomingMessage.emit (events.js:185:7)\n    at endReadableNT (_stream_readable.js:974:12)\n    at _combinedTickCallback (internal/process/next_tick.js:80:11)\n    at process._tickDomainCallback (internal/process/next_tick.js:128:9)"},"message":"all shards failed: [search_phase_execution_exception] all shards failed"}
2018-02-26T21:27:59.229362800Z {"type":"log","@timestamp":"2018-02-26T21:27:59Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":19,"message":"Unable to fetch data from kibana collector"}
2018-02-26T21:27:59.596460100Z Unhandled rejection TypeError: Cannot read property 'result' of undefined
2018-02-26T21:27:59.596499100Z     at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:90:50
2018-02-26T21:27:59.596504200Z     at Array.filter (native)
2018-02-26T21:27:59.596508100Z     at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:90:29
2018-02-26T21:27:59.596512000Z     at next (native)
2018-02-26T21:27:59.596516000Z     at step (/usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:18:202)
2018-02-26T21:27:59.596519700Z     at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:18:383
2018-02-26T21:27:59.596523300Z     at tryCatcher (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/util.js:11:23)
2018-02-26T21:27:59.596526900Z     at Promise._settlePromiseFromHandler (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:489:31)
2018-02-26T21:27:59.596530500Z     at Promise._settlePromise (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:546:18)
2018-02-26T21:27:59.596534100Z     at Promise._settlePromise0 (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
2018-02-26T21:27:59.596537700Z     at Promise._settlePromises (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
2018-02-26T21:27:59.596541200Z     at Promise._fulfill (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)
2018-02-26T21:27:59.596544800Z     at MappingPromiseArray.PromiseArray._resolve (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise_array.js:125:19)
2018-02-26T21:27:59.596548600Z     at MappingPromiseArray._promiseFulfilled (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/map.js:97:18)
2018-02-26T21:27:59.596552200Z     at Promise._settlePromise (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:551:26)
2018-02-26T21:27:59.596555700Z     at Promise._settlePromise0 (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
2018-02-26T21:27:59.596559300Z     at Promise._settlePromises (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
2018-02-26T21:27:59.596562900Z     at Promise._fulfill (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)
2018-02-26T21:27:59.596581900Z     at Promise._settlePromise (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:559:21)
2018-02-26T21:27:59.596585900Z     at Promise._settlePromise0 (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
2018-02-26T21:27:59.596589400Z     at Promise._settlePromises (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
2018-02-26T21:27:59.596593000Z     at Promise._fulfill (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)

When this happens, Kibana instead of logging in Json format, it logs the Unhandled rejection TypeError as normal lines.

1. The bug here seems to be that we need to make sure this is not an unhandled rejection/exception.
2. When this happens, we should use the same logging format that we were using before (eg: json format).

[tsullivan]

Hi, sorry for chiming in so late. Somehow this bug didn't get visibility.

It looks like this was fixed but only in 6.3. The fix should be re-applied and backported for older versions

[damianpfister]

@tsullivan @chrisronline - any ideas what the bug reference is for this issue? I am curious what previous releases the fix will be back-ported to and whether that includes any 5.x releases of Kibana.

[tsullivan]

Hi, sorry that I missed this issue is still open.

The severity seems pretty high since it doesn't look like there are any workarounds. It should be fixed and backported. If it affects 5.x, it should be backported to that.

[chrisronline]

@gmoskovicz Is there a simple way to reproduce this?

[gmoskovicz]

From what i have seen this should be happening when Elasticsearch returns a 503 Status Code.

[wzyonggege]

you might need to check the Health of ElasticSearch

[cachedout]

@chrisronline Given that #5420 was merged, do you feel comfortable with this being closed?

[chrisronline]

Yes! Thanks for the reminder