Add warnings/errors when V2 templates would match same indices as V1 #54367

dakrone · 2020-03-27T21:29:28Z

With the introduction of V2 index templates, we want to warn users that templates they put in place
might not take precedence (because v2 templates are going to "win"). This adds this validation at
PUT time for both V1 and V2 templates with the following rules:

When creating or updating a V2 template

If the v2 template would match indices for an existing v1 template or templates, provide a
warning (through the deprecation logging so it shows up to the client) as well as logging the
warning

The v2 warning looks like:

index template [my-v2-template] has index patterns [foo-*] matching patterns from existing older
templates [old-v1-template,match-all-template] with patterns (old-v1-template =>
[foo*],match-all-template => [*]); this template [my-v2-template] will take
precedence during new index creation

When creating a V1 template

If the v1 template is for index patterns of "*" and a v2 template exists, warn that the v2
template may take precedence
If the v1 template is for index patterns other than all indices, and a v2 template exists that
would match, throw an error preventing creation of the v1 template

When updating a V1 template (without changing its existing `index_patterns`!)

If the v1 template is for index patterns that would match an existing v2 template, warn that the
v2 template may take precedence.

The v1 warning looks like:

template [my-v1-template] has index patterns [*] matching patterns from existing index templates
[existing-v2-template] with patterns (existing-v2-template => [foo*]); this template [my-v1-template] 
may be ignored in favor of an index template at index creation time

And the v1 error looks like:

template [my-v1-template] has index patterns [foo*] matching patterns from existing index templates
[existing-v2-template] with patterns (existing-v2-template => [f*]), use index templates (/_index_template) instead

Relates to #53101

With the introduction of V2 index templates, we want to warn users that templates they put in place might not take precedence (because v2 templates are going to "win"). This adds this validation at `PUT` time for both V1 and V2 templates with the following rules: ** When creating or updating a V2 template - If the v2 template would match indices for an existing v1 template or templates, provide a warning (through the deprecation logging so it shows up to the client) as well as logging the warning The v2 warning looks like: ``` index template [my-v2-template] has index patterns [foo-*] matching patterns from existing older templates [old-v1-template,match-all-template] with patterns (old-v1-template => [foo*],match-all-template => [*]); this template [my-v2-template] will take precedence during new index creation ``` ** When creating a V1 template - If the v1 template is for index patterns of `"*"` and a v2 template exists, warn that the v2 template may take precedence - If the v1 template is for index patterns other than all indices, and a v2 template exists that would match, throw an error preventing creation of the v1 template ** When updating a V1 template (without changing its existing `index_patterns`!) - If the v1 template is for index patterns that would match an existing v2 template, warn that the v2 template may take precedence. The v1 warning looks like: ``` template [my-v1-template] has index patterns [*] matching patterns from existing index templates [existing-v2-template] with patterns (existing-v2-template => [foo*]); this template [my-v1-template] may be ignored in favor of an index template at index creation time ``` And the v1 error looks like: ``` template [my-v1-template] has index patterns [foo*] matching patterns from existing index templates [existing-v2-template] with patterns (existing-v2-template => [f*]), use index templates (/_index_template) instead ``` Relates to elastic#53101

elasticmachine · 2020-03-27T21:30:14Z

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

dakrone · 2020-03-27T21:31:04Z

It's also worth noting to anyone following, that we are planning to be stricter for 8.0+ versions, where creating new v1 templates will not be allowed, only updating existing v1 templates (in favor of doing things with v2 templates). However, this functionality is not yet present.

martijnvg

LGTM 👍 (assuming builds succeed)

martijnvg · 2020-03-30T08:32:19Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java

@@ -272,6 +280,21 @@ static ClusterState addIndexTemplateV2(final ClusterState currentState, final bo
            throw new IllegalArgumentException("index template [" + name + "] already exists");
        }

+        Map<String, List<String>> overlaps = findConflictingV1Templates(currentState, name, template.indexPatterns());
+        if (overlaps.size() > 0) {


I think we should overlapping v2 templates with the same priority in the same way (but then throw an error) (in another change)

Good idea, I'll add a TODO item to add this validation to the meta issue.

martijnvg · 2020-03-30T08:33:40Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java

+     */
+    static Map<String, List<String>> findConflictingV1Templates(final ClusterState state, final String candidateName,
+                                                                final List<String> indexPatterns) {
+        Automaton v2automaton = Regex.simpleMatchToAutomaton(indexPatterns.toArray(Strings.EMPTY_ARRAY));


👍 This method makes validating overlapping templates really easy for index templates v2 :)

martijnvg · 2020-03-30T08:38:34Z

test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java

@@ -549,9 +549,29 @@ private void wipeCluster() throws Exception {
                        adminClient().performRequest(new Request("DELETE", "_template/" + template));
                    }
                }
+                try {
+                    adminClient().performRequest(new Request("DELETE", "_index_template/*"));
+                    adminClient().performRequest(new Request("DELETE", "_component_template/*"));


I would have expected that this change was needed in the change that added component templates and not in this change? (not questioning the need for it, but timing)

So the index and component templates hung around after YAML tests, but because there was never any warnings/errors, and because they aren't actually used for index creation yet, they didn't cause any errors. This is required now because they were hanging around and then the regular template stuff caused errors/warnings.

thanks, makes sense.

dakrone · 2020-03-30T15:02:30Z

@elasticmachine update branch

dakrone · 2020-03-30T15:51:38Z

Jenkins ran out of memory running the PR CI, so I'm restarting the builds that didn't finish since they're wedged now

@elasticmachine run elasticsearch-ci/1
@elasticmachine run elasticsearch-ci/2
@elasticmachine run elasticsearch-ci/bwc
@elasticmachine run elasticsearch-ci/packaging-sample-matrix-windows

…tic#54367) * Add warnings/errors when V2 templates would match same indices as V1 With the introduction of V2 index templates, we want to warn users that templates they put in place might not take precedence (because v2 templates are going to "win"). This adds this validation at `PUT` time for both V1 and V2 templates with the following rules: ** When creating or updating a V2 template - If the v2 template would match indices for an existing v1 template or templates, provide a warning (through the deprecation logging so it shows up to the client) as well as logging the warning The v2 warning looks like: ``` index template [my-v2-template] has index patterns [foo-*] matching patterns from existing older templates [old-v1-template,match-all-template] with patterns (old-v1-template => [foo*],match-all-template => [*]); this template [my-v2-template] will take precedence during new index creation ``` ** When creating a V1 template - If the v1 template is for index patterns of `"*"` and a v2 template exists, warn that the v2 template may take precedence - If the v1 template is for index patterns other than all indices, and a v2 template exists that would match, throw an error preventing creation of the v1 template ** When updating a V1 template (without changing its existing `index_patterns`!) - If the v1 template is for index patterns that would match an existing v2 template, warn that the v2 template may take precedence. The v1 warning looks like: ``` template [my-v1-template] has index patterns [*] matching patterns from existing index templates [existing-v2-template] with patterns (existing-v2-template => [foo*]); this template [my-v1-template] may be ignored in favor of an index template at index creation time ``` And the v1 error looks like: ``` template [my-v1-template] has index patterns [foo*] matching patterns from existing index templates [existing-v2-template] with patterns (existing-v2-template => [f*]), use index templates (/_index_template) instead ``` Relates to elastic#53101 * Remove v2 index and component templates when cleaning up tests * Finish half-finished comment sentence * Guard template removal and ignore for earlier versions of ES Co-authored-by: Elastic Machine <[email protected]>

* Add warnings/errors when V2 templates would match same indices… (#54367) * Add warnings/errors when V2 templates would match same indices as V1 With the introduction of V2 index templates, we want to warn users that templates they put in place might not take precedence (because v2 templates are going to "win"). This adds this validation at `PUT` time for both V1 and V2 templates with the following rules: ** When creating or updating a V2 template - If the v2 template would match indices for an existing v1 template or templates, provide a warning (through the deprecation logging so it shows up to the client) as well as logging the warning The v2 warning looks like: ``` index template [my-v2-template] has index patterns [foo-*] matching patterns from existing older templates [old-v1-template,match-all-template] with patterns (old-v1-template => [foo*],match-all-template => [*]); this template [my-v2-template] will take precedence during new index creation ``` ** When creating a V1 template - If the v1 template is for index patterns of `"*"` and a v2 template exists, warn that the v2 template may take precedence - If the v1 template is for index patterns other than all indices, and a v2 template exists that would match, throw an error preventing creation of the v1 template ** When updating a V1 template (without changing its existing `index_patterns`!) - If the v1 template is for index patterns that would match an existing v2 template, warn that the v2 template may take precedence. The v1 warning looks like: ``` template [my-v1-template] has index patterns [*] matching patterns from existing index templates [existing-v2-template] with patterns (existing-v2-template => [foo*]); this template [my-v1-template] may be ignored in favor of an index template at index creation time ``` And the v1 error looks like: ``` template [my-v1-template] has index patterns [foo*] matching patterns from existing index templates [existing-v2-template] with patterns (existing-v2-template => [f*]), use index templates (/_index_template) instead ``` Relates to #53101 * Remove v2 index and component templates when cleaning up tests * Finish half-finished comment sentence * Guard template removal and ignore for earlier versions of ES Co-authored-by: Elastic Machine <[email protected]> * Also ignore 500 errors when clearing index template v2 templates Co-authored-by: Elastic Machine <[email protected]>

There is a setting in `ESClientYamlSuiteTestCase` under `usually()` that can install a `global` template changing the number of shards for all indices. This can cause warnings when installing v2 templates (see elastic#54367). This adds these as optional warnings so they don't cause failures regardless of whether the global template is installed or not. These warnings can be removed when our internal template usage has been moved to index templates v2 Relates to elastic#53101

There is a setting in `ESClientYamlSuiteTestCase` under `usually()` that can install a `global` template changing the number of shards for all indices. This can cause warnings when installing v2 templates (see #54367). This adds these as optional warnings so they don't cause failures regardless of whether the global template is installed or not. These warnings can be removed when our internal template usage has been moved to index templates v2 Relates to #53101

There is a setting in `ESClientYamlSuiteTestCase` under `usually()` that can install a `global` template changing the number of shards for all indices. This can cause warnings when installing v2 templates (see elastic#54367). This adds these as optional warnings so they don't cause failures regardless of whether the global template is installed or not. These warnings can be removed when our internal template usage has been moved to index templates v2 Relates to elastic#53101

There is a setting in `ESClientYamlSuiteTestCase` under `usually()` that can install a `global` template changing the number of shards for all indices. This can cause warnings when installing v2 templates (see #54367). This adds these as optional warnings so they don't cause failures regardless of whether the global template is installed or not. These warnings can be removed when our internal template usage has been moved to index templates v2 Relates to #53101

dakrone requested review from martijnvg and probakowski March 27, 2020 21:29

dakrone added :Data Management/Indices APIs APIs to create and manage indices and templates v7.8.0 v8.0.0 labels Mar 27, 2020

martijnvg mentioned this pull request Mar 27, 2020

Composable Templates #53101

Closed

39 tasks

dakrone added 3 commits March 27, 2020 16:19

Remove v2 index and component templates when cleaning up tests

e8ba3cf

Finish half-finished comment sentence

cd1d76d

Guard template removal and ignore for earlier versions of ES

f2dbda0

martijnvg approved these changes Mar 30, 2020

View reviewed changes

Merge branch 'master' into itv2-version-conflict-checks

3fcba1d

dakrone merged commit e502b89 into elastic:master Mar 30, 2020

dakrone deleted the itv2-version-conflict-checks branch March 30, 2020 16:42

dakrone mentioned this pull request Mar 30, 2020

[7.x] Add warnings/errors when V2 templates would match same indices… (#54367) #54449

Merged

dakrone mentioned this pull request Mar 31, 2020

Add allowed warnings to index template v2 YAML tests #54535

Merged

dakrone mentioned this pull request Mar 31, 2020

Add allowed warnings to index template v2 YAML tests (#54535) #54541

Merged

pugnascotia added the >enhancement label May 6, 2020

xeraa mentioned this pull request Oct 11, 2020

es early deprecation fail openzipkin/zipkin#3171

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add warnings/errors when V2 templates would match same indices as V1 #54367

Add warnings/errors when V2 templates would match same indices as V1 #54367

dakrone commented Mar 27, 2020

elasticmachine commented Mar 27, 2020

dakrone commented Mar 27, 2020 •

edited

Loading

martijnvg left a comment

martijnvg Mar 30, 2020

dakrone Mar 30, 2020

martijnvg Mar 30, 2020

martijnvg Mar 30, 2020

dakrone Mar 30, 2020

martijnvg Mar 31, 2020

dakrone commented Mar 30, 2020

dakrone commented Mar 30, 2020

Add warnings/errors when V2 templates would match same indices as V1 #54367

Add warnings/errors when V2 templates would match same indices as V1 #54367

Conversation

dakrone commented Mar 27, 2020

When creating or updating a V2 template

When creating a V1 template

When updating a V1 template (without changing its existing index_patterns!)

elasticmachine commented Mar 27, 2020

dakrone commented Mar 27, 2020 • edited Loading

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Mar 30, 2020

Choose a reason for hiding this comment

dakrone Mar 30, 2020

Choose a reason for hiding this comment

martijnvg Mar 30, 2020

Choose a reason for hiding this comment

martijnvg Mar 30, 2020

Choose a reason for hiding this comment

dakrone Mar 30, 2020

Choose a reason for hiding this comment

martijnvg Mar 31, 2020

Choose a reason for hiding this comment

dakrone commented Mar 30, 2020

dakrone commented Mar 30, 2020

When updating a V1 template (without changing its existing `index_patterns`!)

dakrone commented Mar 27, 2020 •

edited

Loading