Allow to redefine the default index when not found

[MarneusCalgarXP]

Kibana version: 5.0

Elasticsearch version: 5.0

Server OS version: Centos 6

Browser version: Chrome 54.0.2840.71 m

Browser OS version: Win 7

Original install method (e.g. download page, yum, from source, etc.): kibana-5.0.0-linux-x86_64.tar.gz

Description of the problem including expected versus actual behavior: After migrating ".kibana" index from kibana 4 to kibana 5, without migrating an index matching the <kibana default index setting>, kibana becomes unusable (only the left bar appears, nothing in the main view).

Steps to reproduce:

1. start an elastic 1.x node
2. create an index "logstash-test-2016.11.10"
3. start a kibana 4.0 plugged to the elastic 1.x node
4. define the default index to "[logstash-test-]YYYY.MM.DD"
5. start an elastic 5.0 node
6. migrate the .kibana index using "index from remote" (or use the migration plugin). DO NOT migrate the index "logstash-test-2016.11.10"
7. start a kibana 5. plugged to the elastic 5.0 node
8. ==> only the left bar appears, nothing in the main view, and error logs in javascript console

Errors in browser console (if relevant):

loading default index pattern 2commons.bundle.js?v=14438:38 Error: IndexPattern's configured pattern does not match any indices at kibana.bundle.js?v=14438:25 at processQueue (commons.bundle.js?v=14438:38) at commons.bundle.js?v=14438:38 at Scope.$eval (commons.bundle.js?v=14438:39) at Scope.$digest (commons.bundle.js?v=14438:39) at Scope.$apply (commons.bundle.js?v=14438:39) at done (commons.bundle.js?v=14438:37) at completeRequest (commons.bundle.js?v=14438:37) at XMLHttpRequest.xhr.onload (commons.bundle.js?v=14438:37)

Provide logs and/or server output (if relevant):

Describe the feature:

Show a view allowing to redefine the default index setting.

Workaround

Currently, the workaround I've found is to redefine this setting in elasticsearch.
ie:
PUT .kibana/config/5.0.0
{
"buildNum": 14438,
"defaultIndex": "another-index"
}

[Bargs]

Are you able to access the Settings page when this happens? If not, what happens when you click the settings link? We had another report of this but it seemed to be a transient issue. I tried to reproduce in master but I'm not getting any error when no indices exist.

[ppf2]

@Bargs I can reproduce this consistently by simulating an upgrade.

1. Have Kibana 4 hook up to a default index pattern referencing an index that exists in ES 2.x (eg. logstash-2016.10.13 default index pattern in Kibana -> logstash-2016.10.13 index)
2. With the above default index pattern saved to the .kibana index in Kibana 4, set up an Elastic Stack 5.0 that is hooked up to a copy of the .kibana index from Step 1 above (to simulate an upgrade). However, do not migrate or create any index named logstash-2016.10.13 in ES 5.0. In order to get Kibana 5 to use the copy of the .kibana index from Kibana 4, delete the .kibana index on ES 5.0 and quickly stop Kibana 5 (to prevent it from recreating it on Kibana 5), then copy the .kibana index in place, restart ES and Kibana 5.
3. When Kibana 5 launches, screens like Dev Tools and Management will all be blank, and if you use Dev Tools in Firefox/Chrome, you will see that this is because of a 404 error showing that it is attempting to get the mapping from logstash-2016.10.13 (which does not exist in the ES 5 cluster).

4 . If I then create a dummy index named logstash-2016.10.13 via curl and then refresh Kibana, the screens start working. Interestingly, if I then delete the logstash-2016.10.13 index and refresh Kibana, everything still works from this point onwards - I think this is what the "transient" observation from other users are coming from. It seems to fix itself after the first successful attempt to get mapping via the default index pattern. But I can certainly see why users upgrading to 5.0 are reporting blank screens when the default index pattern does not map to any existing indices in the 5.0 cluster right after the upgrade.

[Bargs]

Thanks @ppf2 for the detailed steps. I see what the issue is now.

Now that we expect searchable and aggregatable flags on fields in 5.0, we needed a way to update 4.x index patterns that don't have this info. I opted to refresh the fields automatically if that info is missing. This blank page is an unintended side effect. If there are no indices matching the pattern the mapping request fails and the error isn't handled. It also explains the "transient" nature of the issue, the only way to end up with an index pattern without this aggregatable/searchable info is to upgrade from 4.x without migrating your data indices.

So the situation is that the user basically has an invalid 5.0 index pattern, and we can't make it valid until they migrate or create new data indices. I'm thinking the best we can do is display a helpful error and possibly redirect to the index pattern page. What do you think @BigFunger @w33ble @ycombinator ?

[ppf2]

Given that once there is an index in place that matches the pattern, things will start working from that point onwards, catching this condition and throwing an intuitive error to the UI is likely sufficient here, thx!

[w33ble]

I'm thinking the best we can do is display a helpful error and possibly redirect to the index pattern page.

I think both showing a helpful error message and redirecting makes sense. Sounds like there's really nothing else we can do here. The user can either add/migrate data or remove the index pattern from Kibana.

[nuarhu]

I've incidentally managed to somehow configure the wrong indexPattern (new to Kibana). We don't have a logstash-* index in elastic, only specialized other indexes. Now, the Kibana UI seems to be unusable, it doesn't even display the error (only in the JS console) and instead shows a blank page. What kind of user experience is that?

Please don't tell me that instead of handling a JS error correctly and letting the user change the setting that fails to fetch data you want me to add a dummy index to the elastic backend? That sounds wrong in so many ways - what if some online-only admin manages to type in the wrong pattern? Just display that form again where the pattern can be changed.

Sorry for my ranting. Otherwise it's a nice tool (yes, I have seen it work).

[w33ble]

Please don't tell me that instead of handling a JS error correctly and letting the user change the setting that fails to fetch data you want me to add a dummy index to the elastic backend?

No, that's not the resolution. As mentioned above, we need to handle that error and allow the user to do something with the index pattern that was configured.

Adding a dummy index is, however, a workaround for now, until this bug is fixed.

[krychlic1]

I am having the same problem where I deleted filebeat index on accident from kibana and now all the screens are blank. However, here is my error:

Request URL:http://server:5601/elasticsearch/packetbeat-*/_mapping/field/*?_=1487601639728&ignore_unavailable=false&allow_no_indices=false&include_defaults=true
Request Method:GET
Status Code:404 Not Found
Remote Address:server:5601

So, I cannot just create a dummy index because its trying to map to 'packetbeat-*'
Any ideas?

[weltenwort]

Until this is fixed in Kibana, the following query is a pretty safe way to reset the default index pattern for most users:

% curl -XPOST '${ELASTICSEARCH_HOST}/.kibana/_update_by_query?pretty&wait_for_completion&refresh' -H 'Content-Type: application/json' -d'
{
  "script": {
    "inline": "ctx._source.defaultIndex = null",
    "lang": "painless"
  },
  "query": {
    "term": {
      "_type": "config"
    }
  }
}
'

[barryhatfield]

Thanks weltenwort, that resolved my issues with a blank management, dashboard and visualization page. I also had to do the following to resolve a blank discover page:

curl -XPOST 'localhost:9200/_reindex?pretty' -d'
{
  "source": {
    "index": ".kibana"
  },
  "dest": {
    "index": ".kibana1"
  },
  "script": {
    "inline": "ctx._source.remove(\"version\")"
  }
}
'

curl -XPOST 'localhost:9200/_reindex?pretty' -d'
{
  "source": {
    "index": ".kibana1"
  },
  "dest": {
    "index": ".kibana2"
  },
  "script": {
    "inline": "ctx._source.remove(\"hits\")"
  }
}
'