Voting config exclusions should work with absent nodes

[DaveCTurner]

Today the voting config exclusions API accepts node filters and resolves them to a collection of node IDs against the current cluster membership.

This is problematic since we may want to exclude nodes that are not currently members of the cluster. For instance:

• if attempting to remove a flaky node from the cluster you cannot reliably exclude it from the voting configuration since it may not reliably be a member of the cluster

• if cluster.auto_shrink_voting_configuration: false then naively shrinking the cluster will remove some nodes but will leaving their node IDs in the voting configuration. The only way to clean up the voting configuration is to grow the cluster back to its original size (potentially replacing some of the voting configuration) and then use the exclusions API.

I'm opening this to discuss an alternative API that accepts node names and node IDs (but not node filters in general) and which will exclude present and future matching nodes without requiring a matching node to be present right now.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

[sebgl]

elastic/cloud-on-k8s#1986 brings some context on how this impacts ECK.

[DaveCTurner]

The things we would lose with such an API include:

• validating that you didn't make a mistake and exclude a nonexistent node.
• support for excluding nodes by node attributes, e.g. to exclude a whole rack at once.

[DaveCTurner]

We discussed this last week and think we should do it, just based on node name and node ID.

[zacharymorn]

Hi @DaveCTurner I can take a look at this one.

[DaveCTurner]

Thanks @zacharymorn, go for it.

[zacharymorn]

Hi @DaveCTurner ,

Sorry for the delay on this. I had created a PR based on my understanding.

One thing I did notice though is that when we start to allow absent node names and ids, this by definition would also allow incorrect names and ids as per my understanding there's no way to differentiate between the cases. Please let me know if this is expected.

Thanks,
Zach

[DaveCTurner]

Thanks @zacharymorn, yes, as I noted above with this change we necessarily lose the ability to check that the nodes we're excluding aren't merely incorrect.