Add handling in agent cache for server leaf certs

[freddygv]

Description

• Adds a new internally generated and locally managed ACL token for servers to use for management actions. This new ACL token has a corresponding ACL identity and unlimited permissions given that it is generated for server agents.

• Adds a new server certificate manager for requesting a certificate for servers and updating the TLS configurator as needed.

Testing & Reproduction steps

• Manual testing
• Unit testing with multiple test agents

Links

https://go.hashi.co/csl-204-server-cert

PR Checklist

• updated test coverage
• external facing docs updated
• not a security concern

[freddygv]

Hmm will take a look at these failures on Monday

[boxofrad]

This looks great, nice job! 👏🏻 I really enjoyed getting this review request because it nudged me to read the RFC and learn about this feature.

I've left a bunch of mostly non-blocking comments. The only one I think needs to be addressed is how we're hanging onto stale state stores.

[freddygv]

Err the TLS configurator side needs more thinking. I've been trying to fix the test failures but there is a legitimate issue here:

When an autoTLS cert is present we ALWAYS use it as the outbound cert. However, for WANFed via mesh gateways the secondary DCs can't validate the autoTLS server certificates from servers in the primary because the secondaries don't have the Connect CA roots until federation succeeds. This wasn't an issue before because servers never had an autoTLS cert configured.

For server agents we shouldn't use the autoTLS certs for any outbound connections other than peering. But the configurator currently isn't aware of whether it's attached to a server or client agent.

Defaulting to using the manually configured certs for outbound connections would work, but it's a breaking change. To be honest I don't know why we default to autoTLS when a manual cert exists. Seems kind of odd. Also the auto_config docs say These configurations get merged in as defaults with any user-supplied configuration on the client agent able to override them. But I don't see that happening anywhere?

Thoughts @boxofrad ?

[boxofrad]

Ah yes, I remember staring at this for a long while when we switched to a per-listener TLS configuration. 😅

I'm not 100%, but maybe @i0rek remembers? why we always prefer the auto-encrypt/config certificate in outgoing connections. If I had to hazard a guess, I'd say it's because if you're using one of those features, only their automatically provisioned client certificates (signed by the Connect CA) are accepted by the server, not certificates created by consul tls cert create.

I think it'd be ok to break backwards compatibility here just for the gRPC port, as long as we leave the server/multiplexed port alone. Only peering makes outgoing connections on that port, and we've told folks using auto-{encrypt,config} not to upgrade to 1.13 yet, so I don't think it'd impact anyone.