Node repurpose tool #39403

henningandersen · 2019-02-26T16:16:01Z

When a node is repurposed to master/no-data or no-master/no-data, v7.x
will not start (see #37748 and #37347). The elasticsearch-node repurpose
tool can fix this by cleaning up the problematic data.

Unsure if this should be backported to 7.0 too?

Documentation to be done in a follow-up PR.

When a node is repurposed to master/no-data or no-master/no-data, v7.x will not start (see elastic#37748 and elastic#37347). The `elasticsearch repurpose` tool can fix this by cleaning up the problematic data.

elasticmachine · 2019-02-26T16:16:03Z

Pinging @elastic/es-distributed

andrershov

@henningandersen thanks for your PR! I've left several comments, overall looks good already.

server/src/main/java/org/elasticsearch/env/NodeRepurposeCommand.java

server/src/test/java/org/elasticsearch/env/NodeRepurposeCommandTests.java

andrershov · 2019-03-01T13:12:08Z

server/src/test/java/org/elasticsearch/env/NodeRepurposeCommandIT.java

+import static org.mockito.Matchers.contains;
+
+@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST, numDataNodes = 0)
+public class NodeRepurposeCommandIT extends ESIntegTestCase {


Since it's an integration test, I think we need to be able to verify that the document that we have indexed is lost. For example, you may start one master-only node and one data-only node and then repurpose data-only to be master-only or no-master no-data. In this case, when the repurposed node joins the cluster, the index will be re-created but the document is gone.
Also due to randomization, executeRepurposeCommand is a bit hard to follow. I think here we can stop a particular node using nameFilter and explicitly run repurpose tool successfully on this node and unsuccessfully on another node.

Regarding checking for lost data, I added a check that we loose the index completely in f5396b1 , I think that should suffice, do you agree?

Commit 85cf0e6 fixes test to stop one node, run the tool successfully on that node and unsuccessfully on the other node.

I'm not sure how the test works, but after you restart two nodes - they should not form the cluster, because there is no longer the master node. It'd nice to figure out why ensureGreen returns successfully in this case.
But in general, I think, having one dedicated master-eligible node and one data node and repurposing of data node (mentioned in the comment) would be easier to follow.

@andrershov this works because InternalTestCluster maintains the voting configuration in InternalTestCluster.excludeMasters, which takes the stopping node of the voting configuration. This is used by several other tests (is the default for InternalTestCluster), for instance I think GatewayIndexStateIT.testIndexDeletionWhenNodeRejoins depends on the same.

Changed to one master only and one data only node in 7a0712c.

Sorry, the test needed more adjustments, fixed in 4cbf288

I somehow ran the wrong test 100 times ;-)

andrershov · 2019-03-01T13:25:39Z

server/src/test/java/org/elasticsearch/env/NodeRepurposeCommandTests.java

+        assertEquals("Must not touch files", before, after);
+    }
+
+    private long digestPaths() {


I wonder if we already have a similar function implemented somewhere (it takes 40 lines of code here) or probably it makes sense to add it to some util class. Probably @ywelsch can suggest something.

I found none, will await feedback from @ywelsch on this.

I'm not aware of any such existing functionality. There is BufferedChecksumIndexInput in Lucene, but that will not shorten the code here.

server/src/test/java/org/elasticsearch/env/NodeRepurposeCommandIT.java

server/src/main/java/org/elasticsearch/cluster/coordination/NodeToolCli.java

Moved "how to get verbose info" message to after summary message. Test case now randomly creates up to 10 shards.

…ose_tool

Renamed ElasticSearchNodeCommandIT to avoid confusion with repurpose tool that has a separate IT.

Changed NodeRepurposeCommandIT to be more specific about which node it runs the tool on rather than just try all. Fixed NodeToolCli messaging.

Changed NodeRepurposeCommandIT to validate indexes still exists after repurpose and do not exist after repurpose on all nodes.

henningandersen · 2019-03-13T20:08:41Z

Thanks for a thorough review, @andrershov , I believe I addressed all comments, so is ready for another round.

andrershov

Thanks for the changes @henningandersen. I've left a comment about the test and would like to review it again after it's addressed.

Changed NodeRepurposeCommandIT to be less random according to review comments.

henningandersen · 2019-03-14T18:28:02Z

Thanks for the changes @henningandersen. I've left a comment about the test and would like to review it again after it's addressed.

Thanks @andrershov , I modified the test, please have another look.

Fixed NodeRepurposeCommandIT.

henningandersen · 2019-03-15T07:47:03Z

@elasticmachine run elasticsearch-ci/bwc
@elasticmachine run elasticsearch-ci/default-distro

…ose_tool

henningandersen · 2019-03-15T15:59:02Z

@elasticmachine run elasticsearch-ci/packaging-sample

…node_repurpose_tool

…ose_tool

andrershov

@henningandersen I've left several comments to the test, but no need for the next pass.
Since there is no feedback from @yannick on "digest" function, let's have this custom implementation for now.
As for backporting to 7.0, on the one hand, it would be nice to have it in 7.0, but I'm affraid we don't have enough time for documentation. So I think backport to 7.1 is enough.
@yannick what do you think?

andrershov · 2019-03-18T15:55:37Z

server/src/test/java/org/elasticsearch/env/NodeRepurposeCommandIT.java

-        ensureGreen();
-
-        // docs gone
-        assertFalse(client().prepareGet(indexName, "type1", "1").get().isExists());
        assertTrue(indexExists(indexName));


Let's also check here that "prepareGet" will throw NoShardAvailableActionException, because there is no data node to host the shard.

andrershov · 2019-03-18T16:00:56Z

server/src/test/java/org/elasticsearch/env/NodeRepurposeCommandIT.java

        assertTrue(indexExists(indexName));

        logger.info("--> Restarting and repurposing other node");

        internalCluster().stopRandomNode(s -> true);
        internalCluster().stopRandomNode(s -> true);

-        executeRepurposeCommandForOrdinal(noMasterNoDataSettings, indexUUID, 0);
+        executeRepurposeCommandForOrdinal(noMasterNoDataSettings, indexUUID, 0, 0);

        internalCluster().startMasterOnlyNode();


This is not easy to follow. We've repurposed both nodes as non-master and non-data. And now we're starting them as "master" and "data" correspondingly.
I think, that we at least need to add a comment that describes what we're doing here.
Like "now both nodes are coordinating, let's start one of them as a master node and the second one as the data node, this does not require repurpose tool invocation. However, since they were converted to coordinating nodes previously, all data and metadata should be lost and we can verify it".

andrershov · 2019-03-18T16:02:33Z

server/src/test/java/org/elasticsearch/env/NodeRepurposeCommandIT.java

+
+        ensureGreen();
+
+        // indexes gone.


index is gone

Improvements to NodeRepurposeCommandIT.

ywelsch · 2019-03-19T08:02:51Z

Given that #37748 and #37347 are in 7.0, we should try to get this in there as well. I'm sure we will figure out the docs before release.

When a node is repurposed to master/no-data or no-master/no-data, v7.x will not start (see #37748 and #37347). The `elasticsearch repurpose` tool can fix this by cleaning up the problematic data.

Node repurpose tool

73b17b4

When a node is repurposed to master/no-data or no-master/no-data, v7.x will not start (see elastic#37748 and elastic#37347). The `elasticsearch repurpose` tool can fix this by cleaning up the problematic data.

henningandersen added >feature :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) v8.0.0 v7.2.0 labels Feb 26, 2019

henningandersen requested a review from ywelsch February 26, 2019 18:34

ywelsch requested a review from andrershov February 27, 2019 06:50

henningandersen added the v7.0.0 label Feb 27, 2019

andrershov suggested changes Mar 1, 2019

View reviewed changes

henningandersen added 2 commits March 12, 2019 20:39

Node repurpose tool

18df8d7

Moved "how to get verbose info" message to after summary message. Test case now randomly creates up to 10 shards.

Merge remote-tracking branch 'origin/master' into feature_node_repurp…

3bc683a

…ose_tool

ywelsch mentioned this pull request Mar 13, 2019

Make node.master a dynamic setting #10793

Closed

henningandersen added 4 commits March 13, 2019 16:38

Merge remote-tracking branch 'origin/master' into feature_node_repurp…

89849b3

…ose_tool

Node repurpose tool

4df583c

Renamed ElasticSearchNodeCommandIT to avoid confusion with repurpose tool that has a separate IT.

Node repurpose tool

85cf0e6

Changed NodeRepurposeCommandIT to be more specific about which node it runs the tool on rather than just try all. Fixed NodeToolCli messaging.

Node repurpose tool

f5396b1

Changed NodeRepurposeCommandIT to validate indexes still exists after repurpose and do not exist after repurpose on all nodes.

henningandersen requested a review from andrershov March 13, 2019 20:08

andrershov suggested changes Mar 14, 2019

View reviewed changes

Node repurpose tool

7a0712c

Changed NodeRepurposeCommandIT to be less random according to review comments.

henningandersen requested a review from andrershov March 14, 2019 18:28

Node repurpose tool

4cbf288

Fixed NodeRepurposeCommandIT.

henningandersen added 2 commits March 15, 2019 10:29

Merge remote-tracking branch 'origin/master' into feature_node_repurp…

7c12701

…ose_tool

Merge remote-tracking branch 'origin/master' into feature_node_repurp…

6022047

…ose_tool

henningandersen added 2 commits March 18, 2019 11:53

Merge commit '7893aaec0c2faaf202212ad5f4ad0ee2333378d4' into feature_…

dbe27ef

…node_repurpose_tool

Merge remote-tracking branch 'origin/master' into feature_node_repurp…

e9b30c6

…ose_tool

andrershov approved these changes Mar 18, 2019

View reviewed changes

Node repurpose tool

c7be92e

Improvements to NodeRepurposeCommandIT.

henningandersen merged commit bd04b4f into elastic:master Mar 19, 2019

michaelbaamonde added v7.0.0-rc1 and removed v7.0.0 labels Mar 25, 2019

henningandersen mentioned this pull request Mar 27, 2019

Node repurpose tool docs #40525

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Node repurpose tool #39403

Node repurpose tool #39403

henningandersen commented Feb 26, 2019 •

edited

Loading

elasticmachine commented Feb 26, 2019

andrershov left a comment

andrershov Mar 1, 2019

henningandersen Mar 13, 2019

andrershov Mar 14, 2019

henningandersen Mar 14, 2019

henningandersen Mar 14, 2019

henningandersen Mar 14, 2019

andrershov Mar 1, 2019

henningandersen Mar 13, 2019

ywelsch Mar 18, 2019

henningandersen commented Mar 13, 2019

andrershov left a comment

henningandersen commented Mar 14, 2019

henningandersen commented Mar 15, 2019

henningandersen commented Mar 15, 2019

andrershov left a comment

andrershov Mar 18, 2019

andrershov Mar 18, 2019

andrershov Mar 18, 2019

ywelsch commented Mar 19, 2019

Node repurpose tool #39403

Node repurpose tool #39403

Conversation

henningandersen commented Feb 26, 2019 • edited Loading

elasticmachine commented Feb 26, 2019

andrershov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henningandersen commented Mar 13, 2019

andrershov left a comment

Choose a reason for hiding this comment

henningandersen commented Mar 14, 2019

henningandersen commented Mar 15, 2019

henningandersen commented Mar 15, 2019

andrershov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ywelsch commented Mar 19, 2019

henningandersen commented Feb 26, 2019 •

edited

Loading