Allow for a a more graceful transition from plaintext gossip to encrypted gossip

[rboyer]

Currently if you brought up a multi-dc set of wan-joined consul clusters and forgot to enable gossip encryption, the only way to transition to encrypting the gossip causes a temporary rolling gossip split-brain until every server and agent has been bounced.

This patch (which crosses over into hashicorp/memberlist, too) adds two new config settings that work in conjunction with encrypt:

• allow_insecure_incoming_gossip (requires encryption=true)
  • This loosens security when attempting to process gossip messages. It will first try to decrypt the message with the configured keyring, but if that fails it blindly retries assuming it was plaintext.
• prohibit_secure_outgoing_gossip (requires encryption=true)
  • This disables outbound gossip messages from being encrypted, while still allowing the agent to have a keyring.

The rollout order to avoid a crypto-related gossip split-brain is:

1. everybody has encrypt="", allow_insecure_incoming_gossip=false, prohibit_secure_outgoing_gossip=false by default; no encrypted messages
2. incremental switch to: encrypt="KEY", allow_insecure_incoming_gossip=true, prohibit_secure_outgoing_gossip=true; no encrypted messages, but everybody COULD decrypt
3. incremental switch to: encrypt="KEY", allow_insecure_incoming_gossip=true, prohibit_secure_outgoing_gossip=false; some encrypted messages, and everyone can decrypt
4. incremental switch to: encrypt="KEY", allow_insecure_incoming_gossip=false, prohibit_secure_outgoing_gossip=false ; everything is encrypted, no security holes anymore (equivalent to stock encrypted gossip)

This is a pretty edge case need, so I'm fine with this not being merged. If that's so, it would be terrific if someone could point out any issues with the above as I may need to use this in production as a custom build to transition some clusters regardless of a merge.

[slackpad]

Hi @rboyer thanks for the PR! This is a common use case so I think we want something like this. Once we get 0.7.0 out I'll give this some cycles.

[slackpad]

Sorry this didn't make it in for 0.8 :-(