Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] ':qa:mixed-cluster:v7.X.X#mixedClusterTest' failures #55539

Closed
iverase opened this issue Apr 21, 2020 · 3 comments
Closed

[CI] ':qa:mixed-cluster:v7.X.X#mixedClusterTest' failures #55539

iverase opened this issue Apr 21, 2020 · 3 comments
Assignees
@dakrone
Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >test-failure Triaged test failures from CI

Comments

@iverase
Copy link
Contributor

iverase commented Apr 21, 2020 •
edited
Loading

There are a few test using mixed clusters that are failing. I was able to reproduce locally using the following:

./gradlew ':qa:mixed-cluster:v7.7.0#mixedClusterTest' --tests "org.elasticsearch.backwards.IndexingIT"   -Dtests.seed=4C0785959C950BAF   -Dtests.security.manager=true   -Dtests.locale=he   -Dtests.timezone=America/Barbados   -Dcompiler.java=14

The error that seems to be causing this errors is:

[[reason=ALLOCATION_FAILED], at[2020-04-21T14:24:17.813Z], failed_attempts[1], failed_nodes[[FRHbNZRZTQarwsO04vHPIw]], delayed=false, details[failed shard on node [FRHbNZRZTQarwsO04vHPIw]: failed to create index, failure IllegalArgumentException[unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]], allocation_status[no_attempt]], message [failed to create index], failure [IllegalArgumentException[unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]], markAsStale [true]]
»  java.lang.IllegalArgumentException: unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
»       at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:595) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:549) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:176) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:484) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:246) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]

Looking in the repo it seems to be related to this change (which is a breaking change): #55411

examples:

https://gradle-enterprise.elastic.co/s/6ovzg4cjhkab4
https://gradle-enterprise.elastic.co/s/pv5ozkxb3nlxm
@iverase iverase added >test-failure Triaged test failures from CI :Data Management/Indices APIs APIs to create and manage indices and templates labels Apr 21, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

@dakrone dakrone self-assigned this Apr 21, 2020
dakrone added a commit to dakrone/elasticsearch that referenced this issue Apr 21, 2020
@dakrone
Guard adding the index.prefer_v2_templates settings for pre-7.8 nodes
92ff895 
If some of the nodes are pre-7.8 nodes, they may not support the `index.prefer_v2_templates`
setting (since it exists only on 7.8+). This commit makes sure that we only add this setting to the
index's metadata if all of the nodes are 7.8+. Otherwise we get errors like:

```
[ WARN ][o.e.i.c.IndicesClusterStateService] [v7.7.0-3] [test-snapshot-index][2] marking and sending shard failed due to [failed to create index]
  java.lang.IllegalArgumentException: unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:595) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:549) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:176) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:484) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:246) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
```

Relates to elastic#55411
Relates to elastic#53101
Resolves elastic#55539
@dakrone dakrone mentioned this issue Apr 21, 2020
Merged
@dakrone
Copy link
Member

dakrone commented Apr 21, 2020

@iverase thanks for reporting this, it was a real issue. I opened a PR to fix it.

dakrone added a commit that referenced this issue Apr 21, 2020
@dakrone
Guard adding the index.prefer_v2_templates settings for pre-7.8 nodes (…
55ff146 
…#55546)

If some of the nodes are pre-7.8 nodes, they may not support the `index.prefer_v2_templates`
setting (since it exists only on 7.8+). This commit makes sure that we only add this setting to the
index's metadata if all of the nodes are 7.8+. Otherwise we get errors like:

```
[ WARN ][o.e.i.c.IndicesClusterStateService] [v7.7.0-3] [test-snapshot-index][2] marking and sending shard failed due to [failed to create index]
  java.lang.IllegalArgumentException: unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:595) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:549) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:176) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:484) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:246) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
```

Relates to #55411
Relates to #53101
Resolves #55539
@dakrone
Copy link
Member

dakrone commented Apr 21, 2020

This is resolved in #55546 but since it was a 7.x-only PR it didn't automatically close, so I'm closing it now.

@dakrone dakrone closed this as completed Apr 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dakrone dakrone

Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dakrone @elasticmachine @iverase