Detect leader via the delegate

[vishalnayak]

In Vault, autopilot relying on leader's address to detect the ID is opening up a failure mode.

It is possible for raft config and the running nodes to have different addresses. Vault hasn't yet done the piece where the addresses in the raft config gets updated (possibly by re-adding the existing nodes with updated addresses). The main reason for this is that adding a node is not a straight forward ritual and requires unseal keys et al.

Anyways, if the customers restart the node with a different cluster address, since autopilot expects the addresses to match, autopilot will then start erroring out and state API skips returning some servers.

To get around it, the delegate is optionally made to return a IsLeader as part of known servers.

This doesn't affect Consul since the old style leader detection is still in place if the detection via the delegate fails.

[mkeeler]

I think this should be fine but could you add a test similar to this one:

raft-autopilot/state_test.go

Line 84 in f273c7b

func TestGatherNextStateInputs(t *testing.T) {

[mkeeler]

On a meta note, if the raft config doesn't contain the updated address yet then how is raft working at all?

The addresses in the config are used to initiate replication, so it is possible that the leaders address doesn't have to be accurate but all the others will. You may want to consider then how the addresses can be kept in sync to prevent outages when nodes are restarted.

Also there is a distinction between general raft data which is stored via the LogStore and the raft configuration which is stored via the StableStore. I haven't thought it through much but does the stable store need to be guarded by the seal or would it be sufficient to only guard the log store?

[mkeeler]

All thats missing is the test then we could merge this. I still am not sure about the core use case of not being able to update the addresses in raft until after Vault is unsealed but I will defer to the vault team on the viability of the other approach.

[vishalnayak]

Currently when nodes are restarted, Vault expects the same address to be used for nodes that are in the raft config. This fix is for only when the addresses are attempted to be updated during a restart.