Add rolling upgrade multi cluster test module #38277

martijnvg · 2019-02-03T12:45:55Z

This test starts 2 clusters, each with 3 nodes.
First the leader cluster is started and tests are run against it and
the the follower cluster is started and tests execute against this two cluster.

Then the follower cluster is upgraded, one node at a time.
After that the leader cluster is upgraded, one node at a time.
Every time a node is upgraded tests are ran while both clusters are online.
(and either leader cluster has mixed node versions or the follower cluster)

This commit only tests CCR index following, but could be used for CCS tests as well.
In particular for CCR, unidirectional index following is tested during a rolling upgrade.
During the test several indices are created and followed in the leader cluster before or
while the follower cluster is being upgraded.

This tests also verifies that attempting to follow an index in the upgraded cluster
from the not upgraded cluster fails. After both clusters are upgraded following the
index that previously failed should succeed.

Relates to #37231 and #38037

This test starts 2 clusters, each with 3 nodes. First the leader cluster is started and tests are run against it and the the follower cluster is started and tests execute against this two cluster. Then the follower cluster is upgraded, one node at a time. After that the leader cluster is upgraded, one node at a time. Every time a node is upgraded tests are ran while both clusters are online. (and either leader cluster has mixed node versions or the follower cluster) This commit only tests CCR index following, but could be used for CCS tests as well. In particular for CCR, unidirectional index following is tested during a rolling upgrade. During the test several indices are created and followed in the leader cluster before or while the follower cluster is being upgraded. This tests also verifies that attempting to follow an index in the upgraded cluster from the not upgraded cluster fails. After both clusters are upgraded following the index that previously failed should succeed. Relates to elastic#37231 and elastic#38037

elasticmachine · 2019-02-03T12:45:57Z

Pinging @elastic/es-distributed

elasticmachine · 2019-02-04T15:24:26Z

Pinging @elastic/es-core-infra

martijnvg · 2019-02-05T16:28:43Z

This PR is the first step of properly testing ccr during a rolling upgrade. Currently there is only a test that verifies that unidirectional index following works whiling doing a rolling upgrade. As a follow up auto follow patterns should be tested and also bidirection index following should be tested while doing a rolling upgrade (after we decided how we think bi-directional index following should be working during a rolling upgrade).

ywelsch

Thanks for these tests @martijnvg. I've left some smaller comments, looking very good already.

x-pack/qa/rolling-upgrade-multi-cluster/build.gradle

...ti-cluster/src/test/java/org/elasticsearch/upgrades/AbstractMultiClusterUpgradeTestCase.java

...ling-upgrade-multi-cluster/src/test/java/org/elasticsearch/upgrades/CcrRollingUpgradeIT.java

ywelsch · 2019-02-08T16:42:13Z

...ling-upgrade-multi-cluster/src/test/java/org/elasticsearch/upgrades/CcrRollingUpgradeIT.java

+                    // At this point all nodes in both clusters have been updated and
+                    // the leader cluster can now will leader_index4 in the follower cluster:
+                    followIndex(leaderClient(), "follower", "leader_index4", "follower_index4");
+                    assertBusy(() -> verifyTotalHitCount("follower_index4", 64, leaderClient()));


perhaps increase timeout on the assertBusy (same for the other ones in this class).

Do you think the timeouts need to be increased, because CI workers may be slow?
Locally I have not seen this fail, because there wasn't enough time to replicate the documents from leader to follower index.

Yes, I was worrying about slower CI workers. The assertBusy also waits on timed events e.g. on the internal auto-refresh on the follower index. Perhaps we can do an explicit refresh to speed things up?

👍 yes, that makes sense.

...ling-upgrade-multi-cluster/src/test/java/org/elasticsearch/upgrades/CcrRollingUpgradeIT.java

ywelsch · 2019-02-11T09:06:00Z

...ling-upgrade-multi-cluster/src/test/java/org/elasticsearch/upgrades/CcrRollingUpgradeIT.java

+                    ResponseException e = expectThrows(ResponseException.class,
+                        () -> followIndex(leaderClient(), "follower", "leader_index4", "follower_index4"));
+                    assertThat(e.getMessage(), containsString("the snapshot was created with Elasticsearch version ["));
+                    assertThat(e.getMessage(), containsString("] which is higher than the version of this node ["));


we should improve the exception message here /cc @tbrooks8

…luster_tests

…ts own test

martijnvg · 2019-02-11T17:11:30Z

@ywelsch I've updated the PR.

…luster_tests

checking total hitcount inside assertBusy(...)

martijnvg · 2019-02-12T14:06:07Z

I will let this bake on master for a while before backporting.

This test starts 2 clusters, each with 3 nodes. First the leader cluster is started and tests are run against it and then the follower cluster is started and tests execute against this two cluster. Then the follower cluster is upgraded, one node at a time. After that the leader cluster is upgraded, one node at a time. Every time a node is upgraded tests are ran while both clusters are online. (and either leader cluster has mixed node versions or the follower cluster) This commit only tests CCR index following, but could be used for CCS tests as well. In particular for CCR, unidirectional index following is tested during a rolling upgrade. During the test several indices are created and followed in the leader cluster before or while the follower cluster is being upgraded. This tests also verifies that attempting to follow an index in the upgraded cluster from the not upgraded cluster fails. After both clusters are upgraded following the index that previously failed should succeed. Relates to elastic#37231 and elastic#38037

* Add rolling upgrade multi cluster test module (#38277) This test starts 2 clusters, each with 3 nodes. First the leader cluster is started and tests are run against it and then the follower cluster is started and tests execute against this two cluster. Then the follower cluster is upgraded, one node at a time. After that the leader cluster is upgraded, one node at a time. Every time a node is upgraded tests are ran while both clusters are online. (and either leader cluster has mixed node versions or the follower cluster) This commit only tests CCR index following, but could be used for CCS tests as well. In particular for CCR, unidirectional index following is tested during a rolling upgrade. During the test several indices are created and followed in the leader cluster before or while the follower cluster is being upgraded. This tests also verifies that attempting to follow an index in the upgraded cluster from the not upgraded cluster fails. After both clusters are upgraded following the index that previously failed should succeed. Relates to #37231 and #38037 * Filter out upgraded version index settings when starting index following (#38838) The `index.version.upgraded` and `index.version.upgraded_string` are likely to be different between leader and follower index. In the event that a follower index gets restored on a upgraded node while the leader index is still on non-upgraded nodes. Closes #38835

* Add rolling upgrade multi cluster test module (elastic#38277) This test starts 2 clusters, each with 3 nodes. First the leader cluster is started and tests are run against it and then the follower cluster is started and tests execute against this two cluster. Then the follower cluster is upgraded, one node at a time. After that the leader cluster is upgraded, one node at a time. Every time a node is upgraded tests are ran while both clusters are online. (and either leader cluster has mixed node versions or the follower cluster) This commit only tests CCR index following, but could be used for CCS tests as well. In particular for CCR, unidirectional index following is tested during a rolling upgrade. During the test several indices are created and followed in the leader cluster before or while the follower cluster is being upgraded. This tests also verifies that attempting to follow an index in the upgraded cluster from the not upgraded cluster fails. After both clusters are upgraded following the index that previously failed should succeed. Relates to elastic#37231 and elastic#38037 * Filter out upgraded version index settings when starting index following (elastic#38838) The `index.version.upgraded` and `index.version.upgraded_string` are likely to be different between leader and follower index. In the event that a follower index gets restored on a upgraded node while the leader index is still on non-upgraded nodes. Closes elastic#38835

* Add rolling upgrade multi cluster test module (#38277) This test starts 2 clusters, each with 3 nodes. First the leader cluster is started and tests are run against it and then the follower cluster is started and tests execute against this two cluster. Then the follower cluster is upgraded, one node at a time. After that the leader cluster is upgraded, one node at a time. Every time a node is upgraded tests are ran while both clusters are online. (and either leader cluster has mixed node versions or the follower cluster) This commit only tests CCR index following, but could be used for CCS tests as well. In particular for CCR, unidirectional index following is tested during a rolling upgrade. During the test several indices are created and followed in the leader cluster before or while the follower cluster is being upgraded. This tests also verifies that attempting to follow an index in the upgraded cluster from the not upgraded cluster fails. After both clusters are upgraded following the index that previously failed should succeed. Relates to #37231 and #38037 * Filter out upgraded version index settings when starting index following (#38838) The `index.version.upgraded` and `index.version.upgraded_string` are likely to be different between leader and follower index. In the event that a follower index gets restored on a upgraded node while the leader index is still on non-upgraded nodes. Closes #38835

martijnvg · 2019-02-14T13:48:21Z

This has been backported to 7.x, 7.0 and 6.7 banches.

martijnvg added >test Issues or PRs that are addressing/adding tests v7.0.0 :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features v6.7.0 labels Feb 3, 2019

martijnvg assigned ywelsch, jasontedor and dnhatn Feb 3, 2019

martijnvg requested review from dnhatn, jasontedor and ywelsch February 4, 2019 07:25

martijnvg unassigned ywelsch, jasontedor and dnhatn Feb 4, 2019

alpar-t added the :Delivery/Build Build or test infrastructure label Feb 4, 2019

martijnvg mentioned this pull request Feb 5, 2019

Backport: clean up duplicate follow config parameter code change (#37688) #38443

Merged

jasontedor added v8.0.0 and removed v7.0.0 labels Feb 6, 2019

danielmitterdorfer added v7.2.0 and removed v6.7.0 labels Feb 7, 2019

ywelsch suggested changes Feb 11, 2019

View reviewed changes

ywelsch added the v6.7.0 label Feb 11, 2019

martijnvg added 3 commits February 11, 2019 13:42

Merge remote-tracking branch 'es/master' into rolling_upgrade_multi_c…

df34d3c

…luster_tests

iter

80d5fb9

split testing following from upgraded cluster is not supported into i…

9a95d67

…ts own test

ywelsch self-requested a review February 11, 2019 18:14

ywelsch approved these changes Feb 12, 2019

View reviewed changes

martijnvg added 2 commits February 12, 2019 10:51

Merge remote-tracking branch 'es/master' into rolling_upgrade_multi_c…

f6d3dc9

…luster_tests

move assertBusy(...) calls to helper method and refresh prior to

bb93e8b

checking total hitcount inside assertBusy(...)

martijnvg added the v7.0.0 label Feb 12, 2019

martijnvg merged commit f6e8654 into elastic:master Feb 12, 2019

martijnvg added the backport pending label Feb 12, 2019

martijnvg mentioned this pull request Feb 13, 2019

Filter out upgraded version index settings when starting index following #38838

Merged

martijnvg mentioned this pull request Feb 13, 2019

Backport rolling upgrade multi cluster module #38859

Merged

martijnvg mentioned this pull request Feb 14, 2019

Backport rolling upgrade multi cluster module to 6.7 branch #38888

Merged

martijnvg removed the backport pending label Feb 14, 2019

Tim-Brooks mentioned this pull request Feb 21, 2019

Implement CCR bootstrap from remote #35975

Closed

32 tasks

michaelbaamonde added v7.0.0-rc1 and removed v7.0.0 labels Mar 25, 2019

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add rolling upgrade multi cluster test module #38277

Add rolling upgrade multi cluster test module #38277

martijnvg commented Feb 3, 2019

elasticmachine commented Feb 3, 2019

elasticmachine commented Feb 4, 2019

martijnvg commented Feb 5, 2019

ywelsch left a comment

ywelsch Feb 8, 2019

martijnvg Feb 11, 2019

ywelsch Feb 12, 2019

martijnvg Feb 12, 2019

ywelsch Feb 11, 2019

martijnvg commented Feb 11, 2019

martijnvg commented Feb 12, 2019

martijnvg commented Feb 14, 2019

Add rolling upgrade multi cluster test module #38277

Add rolling upgrade multi cluster test module #38277

Conversation

martijnvg commented Feb 3, 2019

elasticmachine commented Feb 3, 2019

elasticmachine commented Feb 4, 2019

martijnvg commented Feb 5, 2019

ywelsch left a comment

Choose a reason for hiding this comment

ywelsch Feb 8, 2019

Choose a reason for hiding this comment

martijnvg Feb 11, 2019

Choose a reason for hiding this comment

ywelsch Feb 12, 2019

Choose a reason for hiding this comment

martijnvg Feb 12, 2019

Choose a reason for hiding this comment

ywelsch Feb 11, 2019

Choose a reason for hiding this comment

martijnvg commented Feb 11, 2019

martijnvg commented Feb 12, 2019

martijnvg commented Feb 14, 2019