Don't let left nodes hold onto their node-id

[hanshasselberg]

Needs hashicorp/memberlist#223 and hashicorp/serf#601.

Fixes #7396.

This PR makes the merge handler skip nodes that properly left the cluster already and it will not complain about conflicting node ids. That could happen eg during a node rename. One of the problems that might arise because of this is that you are running into a conflict when renaming a node.

Afterwards consul members looks like this:

Node  Address          Status  Type    Build     Protocol  DC   Segment
s1    127.0.0.1:8301   alive   server  1.7.0dev  2         dc1  <all>
s2    127.0.0.1:10008  alive   server  1.7.0dev  2         dc1  <all>
s3    127.0.0.1:10015  alive   server  1.7.0dev  2         dc1  <all>
c1    127.0.0.1:10026  failed  client  1.7.0dev  2         dc1  <default>
c2    127.0.0.1:10033  alive   client  1.7.0dev  2         dc1  <default>
c3    127.0.0.1:10040  alive   client  1.7.0dev  2         dc1  <default>
c4    127.0.0.1:10045  alive   client  1.7.0dev  2         dc1  <default>
c5    127.0.0.1:10049  alive   client  1.7.0dev  2         dc1  <default>
cX    127.0.0.1:10026  alive   client  1.7.0dev  2         dc1  <default>

[mkeeler]

LGTM. Although this will need serf/memberlist module updates too. But those others need merging/releasing first. Going to go ahead and approve assuming you are doing those other things.

[aglover-zendesk]

for posterity, I believe this addresses #4879 as well

[hanshasselberg]

@aglover-zendesk I think so too! Good catch!

[ryanmt]

Is there any technical reason why this fix wasn't pulled into 1.6.x releases? I don't see the change in v1.6.10, although it appears to have been pulled into the 1.5 releases.

We've run into this issue on a cluster running 1.6.6 and would love to be able to upgrade to a 1.6 release while we plan to adopt 1.7.