Support reloading certificates

[F21]

FEATURE REQUEST

1. Please describe the feature you are requesting.
   I want to have the ability to reload node and root certificates in order to use hashicorp's Vault as a CA. I want to be able to issue short lived certificates (up to 24 hours) and rotate them when they expire. In terms of retrieving a new certificate, that would be done using a side-car container in kubernetes (which is not a responsibility of cockroach), however, there needs to be some way to notify cockroach that there's a new certificate and to start using it.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   should-have. When using TLS to secure the nodes, there will be a point where certs will expire and will need to be replaced. Operators should be able to do this without taking nodes down.

3. Provide any additional detail on your proposed use case for this feature.
   N/A

[a-robinson]

Hi @F21, thanks for the request! Certificate rotation is something that we've been discussing recently for other reasons as well, so it's good to hear from a user about their needs. It's worth noting that it could be done today in a rolling fashion by restarting each process one at a time, much like a binary upgrade, but I understand that not needing to restart the process would be preferable.

Out of curiosity, are you using a custom sidecar to interact with Vault or is it something widely used? I'm curious if it's something that could be recommended to others to help manage secure deployments on Kubernetes.

cc @mberhault for additional thinking about this

[F21]

I have not deployed cockroach into prod yet, however I've not seen any open source side-car containers in the wild yet. My plan is to build a custom side-car container in the future (if no one else builds one) and use that to rotate certs using Vault.

[mberhault]

moved to #1674

[a-robinson]

@F21 - if you'd like to contribute to the discussion about what this should look like, an RFC was just sent out in #14254 and we'd love your feedback