Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce transport API for cluster bootstrapping #34961

Merged
Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
b01d321
Introduce transport API for cluster bootstrapping
DaveCTurner Oct 29, 2018
c43f31f
Merge branch 'zen2' into 2018-10-29-bootstrap-transport-api
DaveCTurner Oct 30, 2018
0cc2b20
Fix up post-merge
DaveCTurner Oct 30, 2018
fb5c0ff
Rename setters/getters
DaveCTurner Oct 31, 2018
65e60b1
Javadocs
DaveCTurner Oct 31, 2018
592c87e
Rename setter
DaveCTurner Oct 31, 2018
d73638b
Rename getter
DaveCTurner Oct 31, 2018
eb407f4
Rename timeout getter/setter
DaveCTurner Oct 31, 2018
08dd103
Void setters
DaveCTurner Oct 31, 2018
3c3b35a
Javadocs for GetDiscoveredNodesRequest
DaveCTurner Oct 31, 2018
53e2a79
Javadocs for GetDiscoveredNodesResponse
DaveCTurner Oct 31, 2018
47b3718
Move action to cluster:admin
DaveCTurner Oct 31, 2018
5d97e23
Wrap clients for bootstraping in order to support security
DaveCTurner Oct 31, 2018
bdbd600
Rename waitForNodes -> minimumNodeCount
DaveCTurner Nov 2, 2018
4a73b26
Writeable not Streamable
DaveCTurner Nov 5, 2018
4a5083a
Force immutability
DaveCTurner Nov 5, 2018
17e95d8
Remove blank line
DaveCTurner Nov 5, 2018
ca6d0b3
Writeable not Streamable
DaveCTurner Nov 5, 2018
0f7f8fa
Not final
DaveCTurner Nov 5, 2018
48e6096
Change lengths of ID/names
DaveCTurner Nov 5, 2018
55994a0
Use verifyZeroInteractions
DaveCTurner Nov 5, 2018
ef9ae33
Remove draconian validation
DaveCTurner Nov 5, 2018
8b776b3
Writeable not Streamable
DaveCTurner Nov 5, 2018
7de67e2
Reword
DaveCTurner Nov 5, 2018
8c3047c
Imports
DaveCTurner Nov 5, 2018
45cd25e
Message
DaveCTurner Nov 5, 2018
696b71f
Use Mockito
DaveCTurner Nov 5, 2018
f100de4
Message
DaveCTurner Nov 5, 2018
1efbf70
Make use of default timeout
DaveCTurner Nov 5, 2018
d4818d6
Async impl using ListenableFuture
DaveCTurner Nov 6, 2018
2fd41bb
Move setInitialConfiguration method to coordinator and call it on the…
DaveCTurner Nov 6, 2018
4b50388
Don't throw if initial configuration already set
DaveCTurner Nov 6, 2018
1ff3e0f
Tweak
DaveCTurner Nov 6, 2018
1b62782
Extract and use scheduleUnlessShuttingDown
DaveCTurner Nov 6, 2018
0a5a8a8
Register TransportBootstrapClusterAction with TransportService
DaveCTurner Nov 6, 2018
3ee0174
Register GetDiscoveredNodesAction with TransportService
DaveCTurner Nov 6, 2018
759ad07
Don't use AcknowledgedResponse
DaveCTurner Nov 6, 2018
fc2f125
Refactor test initialisation
DaveCTurner Nov 6, 2018
d0a62cb
Rename minimumNodeCount to waitForNodes
DaveCTurner Nov 6, 2018
ad31d90
Merge branch 'zen2' into 2018-10-29-bootstrap-transport-api
DaveCTurner Nov 6, 2018
abcecf6
Merge branch 'zen2' into 2018-10-29-bootstrap-transport-api
DaveCTurner Nov 6, 2018
457cbc7
Use fields not getters
DaveCTurner Nov 7, 2018
144c1a5
Specifically master-eligible nodes
DaveCTurner Nov 7, 2018
00a2be3
Better exceptions
DaveCTurner Nov 7, 2018
e3637c5
AbstractRunnable
DaveCTurner Nov 7, 2018
d3766b4
Use direct executor
DaveCTurner Nov 7, 2018
100ed18
Use ActionListener.wrap
DaveCTurner Nov 7, 2018
2daf1b1
Implement getResponseReader
DaveCTurner Nov 7, 2018
0f25ba2
Javadocs for setInitialConfiguration
DaveCTurner Nov 7, 2018
2d411a5
Add TODO about adding local node to getFoundPeers
DaveCTurner Nov 7, 2018
ad63cee
Per-suite threadpool
DaveCTurner Nov 7, 2018
8688b92
Parallelize bootstrap requests
DaveCTurner Nov 7, 2018
276251e
Allow default of 30 seconds for startup
DaveCTurner Nov 7, 2018
2325d4b
Imports
DaveCTurner Nov 7, 2018
6d05c29
Better messages
DaveCTurner Nov 7, 2018
b957940
Merge branch 'zen2' into 2018-10-29-bootstrap-transport-api
DaveCTurner Nov 8, 2018
7e2b550
TODO fix this
DaveCTurner Nov 8, 2018
eaba4a7
Compile fixes
DaveCTurner Nov 8, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@
import org.apache.logging.log4j.LogManager;
import org.elasticsearch.action.admin.cluster.allocation.ClusterAllocationExplainAction;
import org.elasticsearch.action.admin.cluster.allocation.TransportClusterAllocationExplainAction;
import org.elasticsearch.action.admin.cluster.bootstrap.BootstrapClusterAction;
import org.elasticsearch.action.admin.cluster.bootstrap.GetDiscoveredNodesAction;
import org.elasticsearch.action.admin.cluster.bootstrap.TransportBootstrapClusterAction;
import org.elasticsearch.action.admin.cluster.bootstrap.TransportGetDiscoveredNodesAction;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthAction;
import org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction;
import org.elasticsearch.action.admin.cluster.node.hotthreads.NodesHotThreadsAction;
Expand Down Expand Up @@ -422,6 +426,8 @@ public <Request extends ActionRequest, Response extends ActionResponse> void reg
actions.register(GetTaskAction.INSTANCE, TransportGetTaskAction.class);
actions.register(CancelTasksAction.INSTANCE, TransportCancelTasksAction.class);

actions.register(GetDiscoveredNodesAction.INSTANCE, TransportGetDiscoveredNodesAction.class);
actions.register(BootstrapClusterAction.INSTANCE, TransportBootstrapClusterAction.class);
actions.register(ClusterAllocationExplainAction.INSTANCE, TransportClusterAllocationExplainAction.class);
actions.register(ClusterStatsAction.INSTANCE, TransportClusterStatsAction.class);
actions.register(ClusterStateAction.INSTANCE, TransportClusterStateAction.class);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.action.admin.cluster.bootstrap;

import org.elasticsearch.action.Action;
import org.elasticsearch.common.io.stream.Writeable.Reader;

public class BootstrapClusterAction extends Action<BootstrapClusterResponse> {
public static final BootstrapClusterAction INSTANCE = new BootstrapClusterAction();
public static final String NAME = "cluster:admin/bootstrap_cluster";

private BootstrapClusterAction() {
super(NAME);
}

@Override
public BootstrapClusterResponse newResponse() {
throw new UnsupportedOperationException("usage of Streamable is to be replaced by Writeable");
ywelsch marked this conversation as resolved.
Show resolved Hide resolved
}

@Override
public Reader<BootstrapClusterResponse> getResponseReader() {
return BootstrapClusterResponse::new;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.action.admin.cluster.bootstrap;

import org.elasticsearch.action.ActionRequest;
import org.elasticsearch.action.ActionRequestValidationException;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;

import java.io.IOException;

/**
* Request to set the initial configuration of master-eligible nodes in a cluster so that the very first master election can take place.
*/
public class BootstrapClusterRequest extends ActionRequest {
private final BootstrapConfiguration bootstrapConfiguration;

public BootstrapClusterRequest(BootstrapConfiguration bootstrapConfiguration) {
this.bootstrapConfiguration = bootstrapConfiguration;
}

public BootstrapClusterRequest(StreamInput in) throws IOException {
super(in);
bootstrapConfiguration = new BootstrapConfiguration(in);
}

/**
* @return the bootstrap configuration: the initial set of master-eligible nodes whose votes are counted in elections.
*/
public BootstrapConfiguration getBootstrapConfiguration() {
return bootstrapConfiguration;
}

@Override
public ActionRequestValidationException validate() {
return null;
}

@Override
public void readFrom(StreamInput in) throws IOException {
throw new UnsupportedOperationException("usage of Streamable is to be replaced by Writeable");
}

@Override
public void writeTo(StreamOutput out) throws IOException {
super.writeTo(out);
bootstrapConfiguration.writeTo(out);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.action.admin.cluster.bootstrap;

import org.elasticsearch.action.ActionResponse;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;

import java.io.IOException;

/**
* Response to a {@link BootstrapClusterRequest} indicating that the cluster has been successfully bootstrapped.
*/
public class BootstrapClusterResponse extends ActionResponse {
private final boolean alreadyBootstrapped;

public BootstrapClusterResponse(boolean alreadyBootstrapped) {
this.alreadyBootstrapped = alreadyBootstrapped;
}

public BootstrapClusterResponse(StreamInput in) throws IOException {
super(in);
alreadyBootstrapped = in.readBoolean();
}

/**
* @return whether this node already knew that the cluster had been bootstrapped when handling this request.
*/
public boolean getAlreadyBootstrapped() {
return alreadyBootstrapped;
}

@Override
public void readFrom(StreamInput in) throws IOException {
throw new UnsupportedOperationException("usage of Streamable is to be replaced by Writeable");
}

@Override
public void writeTo(StreamOutput out) throws IOException {
super.writeTo(out);
out.writeBoolean(alreadyBootstrapped);
}

@Override
public String toString() {
return "BootstrapClusterResponse{" +
"alreadyBootstrapped=" + alreadyBootstrapped +
'}';
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.action.admin.cluster.bootstrap;

import org.elasticsearch.ElasticsearchException;
import org.elasticsearch.cluster.ClusterState.VotingConfiguration;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;
import org.elasticsearch.common.io.stream.Writeable;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;

public class BootstrapConfiguration implements Writeable {

private final List<NodeDescription> nodeDescriptions;

public BootstrapConfiguration(List<NodeDescription> nodeDescriptions) {
if (nodeDescriptions.isEmpty()) {
throw new IllegalArgumentException("cannot create empty bootstrap configuration");
}
this.nodeDescriptions = Collections.unmodifiableList(new ArrayList<>(nodeDescriptions));
}

public BootstrapConfiguration(StreamInput in) throws IOException {
nodeDescriptions = Collections.unmodifiableList(in.readList(NodeDescription::new));
assert nodeDescriptions.isEmpty() == false;
}

public List<NodeDescription> getNodeDescriptions() {
return nodeDescriptions;
}

public VotingConfiguration resolve(Iterable<DiscoveryNode> discoveredNodes) {
final Set<DiscoveryNode> selectedNodes = new HashSet<>();
for (final NodeDescription nodeDescription : nodeDescriptions) {
final DiscoveryNode discoveredNode = nodeDescription.resolve(discoveredNodes);
if (selectedNodes.add(discoveredNode) == false) {
throw new ElasticsearchException("multiple nodes matching {} in {}", discoveredNode, this);
}
}

final Set<String> nodeIds = selectedNodes.stream().map(DiscoveryNode::getId).collect(Collectors.toSet());
assert nodeIds.size() == selectedNodes.size() : selectedNodes + " does not contain distinct IDs";
return new VotingConfiguration(nodeIds);
}

@Override
public void writeTo(StreamOutput out) throws IOException {
out.writeList(nodeDescriptions);
}

@Override
public String toString() {
return "BootstrapConfiguration{" +
"nodeDescriptions=" + nodeDescriptions +
'}';
}

@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
BootstrapConfiguration that = (BootstrapConfiguration) o;
return Objects.equals(nodeDescriptions, that.nodeDescriptions);
}

@Override
public int hashCode() {
return Objects.hash(nodeDescriptions);
}

public static class NodeDescription implements Writeable {

@Nullable
private final String id;

private final String name;

@Nullable
public String getId() {
return id;
}

public String getName() {
return name;
}

public NodeDescription(@Nullable String id, String name) {
this.id = id;
this.name = Objects.requireNonNull(name);
}

public NodeDescription(DiscoveryNode discoveryNode) {
this(discoveryNode.getId(), discoveryNode.getName());
}

public NodeDescription(StreamInput in) throws IOException {
this(in.readOptionalString(), in.readString());
}

@Override
public void writeTo(StreamOutput out) throws IOException {
out.writeOptionalString(id);
out.writeString(name);
}

@Override
public String toString() {
return "NodeDescription{" +
"id='" + id + '\'' +
", name='" + name + '\'' +
'}';
}

public DiscoveryNode resolve(Iterable<DiscoveryNode> discoveredNodes) {
DiscoveryNode selectedNode = null;
for (final DiscoveryNode discoveredNode : discoveredNodes) {
assert discoveredNode.isMasterNode() : discoveredNode;
if (discoveredNode.getName().equals(name)) {
if (id == null || id.equals(discoveredNode.getId())) {
if (selectedNode != null) {
throw new ElasticsearchException(
"discovered multiple nodes matching {} in {}", this, discoveredNodes);
}
selectedNode = discoveredNode;
} else {
throw new ElasticsearchException("node id mismatch comparing {} to {}", this, discoveredNode);
}
} else if (id != null && id.equals(discoveredNode.getId())) {
throw new ElasticsearchException("node name mismatch comparing {} to {}", this, discoveredNode);
}
}
if (selectedNode == null) {
throw new ElasticsearchException("no node matching {} found in {}", this, discoveredNodes);
}

return selectedNode;
}

@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
NodeDescription that = (NodeDescription) o;
return Objects.equals(id, that.id) &&
Objects.equals(name, that.name);
}

@Override
public int hashCode() {
return Objects.hash(id, name);
}
}
}
Loading