Refresh mappings automatically

[segalziv]

When indexing new fields in elasticsearch, kibana doesn't displays the new fields (except from in the _source field). the way to 'tell' kibana that there are new fields in the index is by going into settings-->specific-index-->reload-field-list.
I believe that this should be done automatically. kibana should reload the index mapping and tell that there are new fields available and display them.

[rabbbit]

+1 - I just spent ~5 hours trying to understand why some fields are missing.

Automatic check, periodical check, or a warning would be great.

[rashidkpc]

This is done purposefully for now. We cache the mapping because large mappings are expensive to parse. We cache the normalized mapping to keep things speedy. We're looking into methods of offering a warning.

[jimmyjones2]

Same as #1713?

[segalziv]

@jimmyjones2 I don't think it's exactly the same (although pretty similar), cause currently the new fields are displayed in _source field, but just not as a separate fields. in both cases, reloading field data manually through settings solves this.

[rfarley3]

+1

I'm hitting this wall as well. I'm wondering how to approach this.

The choke point is the refreshFields() js function. It does not have an external facing (non-browser) means to call it. The function is a wrapper for doing a GET then POST to ES, and involves a lot of overhead (ES parses and processes the request). Having ES update kibana upon new mappings seems a bit counter to the design schema (kibana knows about ES, but ES doesn't care if kibana exists). Since kibana will not receive 'refresh' commands from ES, kibana would have to do periodic refreshes which would get computationally expensive. This has left me putting the burden of refreshing onto the program indexing new data; when it handles new doc_types it can query/compare ES mappings to the .kibana cache update if needed. Contact me if you would like a python script that refreshes the .kibana cache.

I'd like to brainstorm some possibilities and help implement an enhancement.

Perhaps a means to know when a search produces results where not all fields are in the cache mapping? This happens in two places:

• On the discover pane, fields with missing index-pattern-fields show hazard icons and display a warning. Perhaps when any of these occur the page prompts 'We noticed some missing index-pattern-fields, would you like to refresh your fields?' With a config option to make this automatic.
• In dashboards, if visualizations use fields with missing index-pattern-fields a warning is shown. The warning could be a link to refresh the fields for that visualization's index-pattern. With a config option to make this automatic.

Ultimately, I would also like to hear from any kibana developers about the possibility to remove the need for any expensive parsing when creating the cache. If I understand things correctly the only parsing done in refreshFields is:

• Converting field type:
  • type=long are cached as type=number
  • _* fields are cached as type=string
• Creating the analyzed and indexed fields from index:
  • if index == analyzed then cache analyzed=true
  • if index != “no” then cache indexed=true
• Append all mappings with count=0, scripted=false
  • I'm not sure when scripted would be true, I've yet to observe this
    This discussion may be for another location, but why are these needed? I don't mean to ruffle feathers. What value do they provide? For instance, is there legacy code with conditionals that require analyzed and indexed to be bools instead of indexed to be an enum? Is there a need to pre-convert type long to number? etc.

[naisanza]

+1 for a check box option to allow auto-reload of field names

[ayashjorden]

Hi all,
Does this issue will also handle indices with VERY LARGE quantity of fields?
For example, we're parsing cookies and query params into fields. this got us to have indices with about 13K fields. when loading Kibana, the fields list only shows little above 200 fields and only when I refresh it, all of the fields apear. this is also causing the Chrome tab that holds Kibana to consume about 2.5GB of memory.

The main issue is that fields that are present in the index template(not the parsed cookie/query-params fields) are not searchable in Kibana until I refresh the list.

Is there a way to tell KIbana what fields it must load for certain index patterns?

Thanks,
Yarden

[megastef]

+1 I like the "check box option to allow auto-reload of field names" - but we automate many things and we would be happy with a server route - or a script! like

refresh_index.sh https://host:9200/kibanaindex optionalIndexPatternName

The advantage of a script is that the backend could trigger it when it is required (e.g. new data indexed)
What in detail needs to be done? Read mapping and write some new doc to .kibana?
Could you point me to the current source code/implementation please?

[eraffenne]

+1 for automatic refresh. I'd like to have a configuration parameter to specify the refreshing rate instead of having to do it manually.

[khatekar]

+1 for automatic refresh.

[jdgonzalez-keedio]

+1 !!!

[ndopico]

+1

[jjmleiro]

+1

[jucaf]

+1

[rodriguezsergio]

Any update for this?

[rfarley3]

I made a python based interface that has a feature that may help people facing this issue: https://github.com/rfarley3/Kibana

[naisanza]

Awesome!

On September 22, 2015 3:10:32 PM EDT, R Farley [email protected] wrote:

I made a python based interface that has a feature that may help people
facing this issue: https://github.com/rfarley3/Kibana

---

Reply to this email directly or view it on GitHub:
#2236 (comment)

[hrishikeshtak]

@rfarley3: Thanks

[msn]

+1 for integrating support for auto refresh in to stock kibana please!

[catalinvr]

+1

[rashidkpc]

Replacing with #6498

[kaihil]

+1

[matusmarcel]

+1
but also I understand auto-reload may be bottleneck. What I don't understand is, why there is not an API for this. We have static mapping, so we know exactly when we change it. And we always change and deploy it automatically. In that case, it would be super efficient just to have few API calls, to handle it.

[jerome83136]

+1 for automatic fields reloading

[Stono]

+1 too

[Zexuz]

+1

[ileaving]

+1

[eric-wei1990]

+1

[attoken]

+1

[apomarinov]

+1 Is there an endpoint that can be hit to do this from PHP?

[andrx]

+1

[wsx8308041]

+1

[mattkime]

resolved via #82223 and will be released as part of 7.11