Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discovery autocomplete feature cause fielddata of indices significant increase #29146

Closed
LoadingZhang opened this issue Jan 23, 2019 · 5 comments
Labels
Feature:KQL KQL impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort

Comments

@LoadingZhang
Copy link

Kibana version: 6.4.0
Elasticsearch version: 6.4.0
Describe the bug:
There are 16TB+/150+Bil indices in ES, also on fields with very high cardinality such as uid.
If turn on autocomplete feature, then input uid: in search bar, kibana would send this query to ES:

{
  "size": 0,
  "timeout": "1s",
  "terminate_after": 100000,
  "query": {
    "match_all": {
      "boost": 1.0
    }
  },
  "aggregations": {
    "suggestions": {
      "terms": {
        "field": "uid",
        "size": 10,
        "shard_size": 10,
        "min_doc_count": 1,
        "shard_min_doc_count": 0,
        "show_term_doc_count_error": false,
        "execution_hint": "map",
        "order": [
          {
            "_count": "desc"
          },
          {
            "_key": "asc"
          }
        ],
        "include": ".*"
      }
    }
  }
}

It will load 160GB+ fielddata to memory, then ES OOM.
Can we have better way to implement this suggestion function to avoid load too much fielddata memory?

@cjcenizal cjcenizal added bug Fixes for quality problems that affect the customer experience Team:Visualizations Visualization editors, elastic-charts and infrastructure labels Feb 12, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app

@timroes timroes added Feature:KQL KQL and removed bug Fixes for quality problems that affect the customer experience labels Feb 13, 2019
@LoadingZhang
Copy link
Author

this problem may be solved by this issue, thanks

@timroes
Copy link
Contributor

timroes commented Feb 7, 2020

@Bargs I am reopening this for further discussion, since there are reports this is not yet fixed. I am wondering why we're sending execution_hint: map, since the documentation states, that it should mainly be used when only a few documents match the query. For our KQL auto complete isn't it most often the opposite?

@timroes timroes reopened this Feb 7, 2020
@timroes timroes added Team:AppArch and removed Team:Visualizations Visualization editors, elastic-charts and infrastructure labels Feb 20, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-arch (Team:AppArch)

@exalate-issue-sync exalate-issue-sync bot added impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort labels Jun 2, 2021
@lukasolson
Copy link
Member

Closing this as resolved due to now using _terms_enum API for autocomplete suggestions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:KQL KQL impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort
Projects
None yet
Development

No branches or pull requests

6 participants