[Discuss][Data Views] Field popularity performance improvement

[streamich]

Index Pattern service discussion mentions that field popularity (or "count") is used only in Discover:

Seems to only be used in discover to sort some fields to the top of the list

Now index pattern is saved every time a user does some field related operation in Discover. Like pressing the "add" or "remove" button on the field. Customer has suggested to improve how field popularity is tracked:

Should popularity be stored in elasticsearch? I think it might be better to manage it on local storage for now If concurrent users are not considered as now.

Even if it should be stored in elasticsearch as-is for sharing popularity, it doesn't seem desirable to update a full index pattern each time. How about making dedicated API for updating popularity, keep it temporarily in memory, and update it periodically?

What do we want to do about field popularity?

• Leave as is, but suppress warnings on concurrent updates. [Feature request] Suppress UI warning when popularizing a field #52266
• Abandon them completely and rethink Index Pattern service. [DISCUSS] Rethinking index patterns; improvements #35481
• Optimize how field popularity is tracked, don't save index pattern on every change. Maybe keep field popularity in a separate saved object or track it locally on client.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[lukeelmers]

Leave as is, but suppress warnings on concurrent updates. #52266

++ I think this is a great way to minimize confusion while we sort out a longer-term strategy.

Abandon them completely and rethink Index Pattern service. #35481

In the past we have discussed the idea of a more thorough "field statistics" service offered as a part of index patterns. AFAIK the design of such a service hasn't been discussed in much detail yet, but presumably would replace the existing indexPattern.popularizeField() method.

Also FWIW, field popularity feels thematically related to the long-standing discussion around usage metrics in Kibana. If a feature like that were eventually implemented, then we could consider rolling field popularity tracking into it as another option. (If it is, in fact, something that users want/need).

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[lizozom]

I recently noticed that when authoring runtime fields and having discover open in another tab, saving runtime fields would occasionally fail and require a refresh, because the index pattern is out of sync. This explains why this happens.

Probably worth considering that usecase as well when thinking of a solution.

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.