Isolated WAN pools

[jsternberg]

We've been using Consul decently successfully for pulling together our infrastructure. Unfortunately, the infrastructure we have has a lot of isolation and certain sections aren't able to talk to others (by design). We've found this leads to a problem with the gossip protocol.

Consider the following infrastructure where there are 3 subnets: A, B, and C. A and B can communicate along with A and C, but B and C are isolated from each other and cannot communicate.

When this is the case, C seems to report that B is dead to A. A then boots B out, but then B rejoins by himself after being told he's kicked out. This leads to intermittent network issues that rely on forwarding datacenters.

What I'd like it a way to isolate the WAN pools. I want A and B to form their own unique WAN pool. If I ask A about B, then he'll forward the request. If I ask B about A, he'll forward the request. If I ask B about C, he won't know what I'm talking about. A never communicates any information about C to B.

With this configuration, it would be possible to have a central hub in A that can communicate with all of them, but doesn't mix them.

Maybe something like consul join -wan -isolate A.

[armon]

This is an issue we are aware of, but presents a number of complex challenges. Our gossip protocol assumes a fully connected network, so more complex setups (hub and spoke, chained, etc) cause issues like this. We are trying to think of better ways to support it, but almost all will require a deep change to the underlying protocols.

[jsternberg]

Is this a problem with the Serf library or with the underlying Raft protocol? I'd like to take a look at it myself if possible, although I can't dedicate too much time to it.

[ryanuber]

@jsternberg this involves both serf and the even lower-level memberlist package. As @armon mentioned the changes required will be deep and involved, so before starting any work it would be best to start brainstorming in a Google Docs document about the design, as there are certainly a very large number of considerations to make here.

[jsternberg]

This feature has become essential for something that we want to do in our infrastructure where previously we were able to work around it. I've been exploring the memberlist code and I think I have an idea about the design.

For the Google Doc in exploring the solution for this, who should I make sure to include when I share the document?

[armon]

Please include me (@armon), @ryanuber and @slackpad.

[slackpad]

Closing this out as a duplicate of the newer #1871.