Use V2 index templates during index creation #54669
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit changes our index creation code to use (and favor!) V2 index templates during index
creation. The creation precedence goes like so, in order of precedence:
- Existing source `IndexMetadata` - for example, when recovering from a peer or a shrink/split/clone
where index templates should not be applied
- A matching V2 index template, if one is found
- When a V2 template is found, all component templates (in the `composed_of` field) are applied
in the order that they appear, with the index template having the 2nd highest precedence (the
create index request always has the top priority when it comes to index settings)
- All matching V1 templates (the old style)
This also adds index template validation when `PUT`-ing a new v2 index template (because this was
required) and ensures that all index and component templates specify *no* top-level mapping type (it
is automatically added when the template is added to the cluster state).
This does not yet implement fine-grained component template merging of mappings, where we favor
merging only a single field's configuration, that will be done in subsequent work.
This also keeps the existing hidden index behavior present for v1 templates, where a hidden index
will match v2 index templates unless they are global (`*`) templates.
Relates to elastic#53101
dakrone
added
:Data Management/Indices APIs
|
Pinging @elastic/es-core-features (:Core/Features/Indices APIs) |
It matches all templates otherwise, so causes warnings
|
@elasticmachine run elasticsearch-ci/2 |
|
Test suite timed out due to Github issues (which are hopefully resolved) @elasticmachine run elasticsearch-ci/2 |
martijnvg
approved these changes
Apr 3, 2020
| @@ -307,7 +307,7 @@ static boolean resolvePipelines(final DocWriteRequest<?> originalRequest, final | |||
| } | |||
| } else if (indexRequest.index() != null) { | |||
| // the index does not exist yet (and this is a valid request), so match index templates to look for pipelines | |||
| List<IndexTemplateMetadata> templates = MetadataIndexTemplateService.findTemplates(metadata, indexRequest.index(), null); | |||
| List<IndexTemplateMetadata> templates = MetadataIndexTemplateService.findV1Templates(metadata, indexRequest.index(), null); | |||
There was a problem hiding this comment.
Should we also check v2 templates here, if default/final pipeline has been specified? If so then I think it is fine to do this in a follow up pr.
| final List<IndexTemplateMetadata> matchedTemplates = findTemplates(metadata, rolloverIndexName, isHidden); | ||
| static void checkNoDuplicatedAliasInIndexTemplate(Metadata metadata, String rolloverIndexName, String rolloverRequestAlias, | ||
| @Nullable Boolean isHidden) { | ||
| final List<IndexTemplateMetadata> matchedTemplates = findV1Templates(metadata, rolloverIndexName, isHidden); |
There was a problem hiding this comment.
Should we check v2 templates here too? If so, then I think it is ok to do this in a followup change as well.
There was a problem hiding this comment.
Definitely, I'll be addressing these in followups!
| @@ -295,12 +320,41 @@ static ClusterState addIndexTemplateV2(final ClusterState currentState, final bo | |||
| deprecationLogger.deprecated(warning); | |||
| } | |||
|
|
|||
| // TODO: validation of index template | |||
There was a problem hiding this comment.
Maybe leave the todo? I think we still need to validate whether templates with the same priority exist that have overlapping patterns?
There was a problem hiding this comment.
This todo was for validating the mappings and settings in an index template (which this PR adds), we have a TODO item in the list on #53101 to make sure explicitly add the check for overlapping v2 template, so no need to worry!
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this pull request
Apr 3, 2020
* Use V2 index templates during index creation
This commit changes our index creation code to use (and favor!) V2 index templates during index
creation. The creation precedence goes like so, in order of precedence:
- Existing source `IndexMetadata` - for example, when recovering from a peer or a shrink/split/clone
where index templates should not be applied
- A matching V2 index template, if one is found
- When a V2 template is found, all component templates (in the `composed_of` field) are applied
in the order that they appear, with the index template having the 2nd highest precedence (the
create index request always has the top priority when it comes to index settings)
- All matching V1 templates (the old style)
This also adds index template validation when `PUT`-ing a new v2 index template (because this was
required) and ensures that all index and component templates specify *no* top-level mapping type (it
is automatically added when the template is added to the cluster state).
This does not yet implement fine-grained component template merging of mappings, where we favor
merging only a single field's configuration, that will be done in subsequent work.
This also keeps the existing hidden index behavior present for v1 templates, where a hidden index
will match v2 index templates unless they are global (`*`) templates.
Relates to elastic#53101
dakrone
added a commit
that referenced
this pull request
Apr 3, 2020
* Use V2 index templates during index creation
This commit changes our index creation code to use (and favor!) V2 index templates during index
creation. The creation precedence goes like so, in order of precedence:
- Existing source `IndexMetadata` - for example, when recovering from a peer or a shrink/split/clone
where index templates should not be applied
- A matching V2 index template, if one is found
- When a V2 template is found, all component templates (in the `composed_of` field) are applied
in the order that they appear, with the index template having the 2nd highest precedence (the
create index request always has the top priority when it comes to index settings)
- All matching V1 templates (the old style)
This also adds index template validation when `PUT`-ing a new v2 index template (because this was
required) and ensures that all index and component templates specify *no* top-level mapping type (it
is automatically added when the template is added to the cluster state).
This does not yet implement fine-grained component template merging of mappings, where we favor
merging only a single field's configuration, that will be done in subsequent work.
This also keeps the existing hidden index behavior present for v1 templates, where a hidden index
will match v2 index templates unless they are global (`*`) templates.
Relates to #53101
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit changes our index creation code to use (and favor!) V2 index templates during index
creation. The creation precedence goes like so, in order of precedence:
IndexMetadata- for example, when recovering from a peer or a shrink/split/clonewhere index templates should not be applied
composed_offield) are appliedin the order that they appear, with the index template having the 2nd highest precedence (the
create index request always has the top priority when it comes to index settings)
This also adds index template validation when
PUT-ing a new v2 index template (because this wasrequired) and ensures that all index and component templates specify no top-level mapping type (it
is automatically added when the template is added to the cluster state).
This does not yet implement fine-grained component template merging of mappings, where we favor
merging only a single field's configuration, that will be done in subsequent work.
This also keeps the existing hidden index behavior present for v1 templates, where a hidden index
will match v2 index templates unless they are global (
*) templates.Relates to #53101