Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose if a field is a metadata field in the field capabilities response #69977

Merged
merged 21 commits into from
Mar 30, 2021

Conversation

jimczi
Copy link
Contributor

@jimczi jimczi commented Mar 4, 2021

This change exposes for each field in the _field_caps response if the field is a metadata field.
This is needed for consumers of this API that want to filter these fields. Currently ML keeps a static list
and QL checks that the family type starts with _. In order to ease the addition of new metadata fields, this
change reworks the strategy in this solution and now only checks for the new flag.
Note that the new flag is also applied at the coordinator level in a best-effort to apply the logic on older nodes
in a mixed-version cluster.

This change exposes for each field in the _field_caps response if the field is a metadata field.
This is needed for consumers of this API that want to filter these fields. Currently ML keeps a static list
and QL checks that the family type starts with `_`. In order to ease the addition of new metadata fields, this
change reworks the strategy in this solution and now only checks for the new flag.
Note that the new flag is also applied at the coordinator level in a best-effort to apply the logic on older nodes
in a mixed-version cluster.
@jimczi jimczi added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types v8.0.0 v7.13.0 labels Mar 4, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Mar 4, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@jimczi
Copy link
Contributor Author

jimczi commented Mar 4, 2021

\cc @elastic/ml-core and @elastic/es-ql

@jimczi jimczi requested a review from markharwood March 4, 2021 15:40
@dimitris-athanasiou
Copy link
Contributor

Awesome! Looks good from ML standpoint.

Copy link
Contributor

@markharwood markharwood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not had a chance to run it yet but added a couple of comments

@astefan
Copy link
Contributor

astefan commented Mar 15, 2021

Nice improvement @jimczi . Indeed, QL filters out any fields starting with "_" and those potentially being added by mapper_size plugin. Thank you for simplifying this!

Copy link
Member

@javanna javanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that I had an in-flight review from a few days ago, that I forgot to submit. super minor things. I am happy with the updates, thanks!

@jimczi
Copy link
Contributor Author

jimczi commented Mar 25, 2021

ci failed on a known issue

@jimczi jimczi closed this Mar 25, 2021
@jimczi jimczi reopened this Mar 25, 2021
@jimczi
Copy link
Contributor Author

jimczi commented Mar 25, 2021

@elasticmachine run elasticsearch-ci/1

Copy link
Member

@javanna javanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mostly ready, I left a couple of questions on naming and the versioning logic, LGTM otherwise.

@jimczi jimczi merged commit fa88a46 into elastic:master Mar 30, 2021
@jimczi jimczi deleted the field_caps_is_metadata branch March 30, 2021 10:13
jimczi added a commit to jimczi/elasticsearch that referenced this pull request Mar 30, 2021
…nse (elastic#69977)

This change exposes for each field in the _field_caps response if the field is a metadata field.
This is needed for consumers of this API that want to filter these fields. Currently ML keeps a static list
and QL checks that the family type starts with `_`. In order to ease the addition of new metadata fields, this
change reworks the strategy in this solution and now only checks for the new flag.
Note that the new flag is also applied at the coordinator level in a best-effort to apply the logic on older nodes
in a mixed-version cluster.
jimczi added a commit to jimczi/elasticsearch that referenced this pull request Mar 30, 2021
jimczi added a commit that referenced this pull request Mar 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team v7.13.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants