Mapping limited to last 5 indices

[jimmyjones2]

Usecase: searching system logs from systemd journal (using the JSON output)

Discover says I have some fields without a cached mapping... turns out components/index_patterns/_mapper.js limits the mapping to the last 5 indices that match the pattern. In my case I only ran libvirt last month, which is more than 5 daily indicies ago, so the field LIBVIRT_DOMAIN (and others) are not available so there is no way for me to visualize them.

Would rather not have to figure out the systemd schema and add a static mapping...

[rashidkpc]

Yeah, this is sort of a challenge, we can expand the number of indices we look at, but that causes the returned response to grow, causing issues with larger mappings. The real solution is creating a way for elasticsearch to tell us about all of the indices that match the pattern, but to reduce the response on the elasticsearch side to just the stuff we need to know.

[jimmyjones2]

@rashidkpc How about using the hash of the mapping to determine if its changed? I've got a monster mapping over lots of indices, if I get them all from elasticsearch at once takes less than a second to return the 5MB mapping (I presume elasticsearch has it cached somewhere). Should be quick for Kibana to hash this response to figure out if it needs to parse the whole thing. Obviously the first time it would need to parse them all, but with a progress bar I don't think people would mind.

[zhaakhi]

A workaround is to change this line in index.js:

return existing.matches.slice(-5); // Grab the most recent 5

to e.g.:

return existing.matches.slice(-100);

You can do this client-side in e.g. the Chrome debugger without changing files on disk

[jimmyjones2]

Superseded by #6498