-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Raft TLS Helm examples #12982
Add Raft TLS Helm examples #12982
Conversation
Co-authored-by: Pascal Reeb <[email protected]>
Hello @in0rdr - Thank you for your contribution! After reviewing the new document you wrote, I have some questions and feedback: We followed along with the doc by starting with the two examples linked at the top (build cluster, create TLS certificate). We aren't quite sure what led to the error you described after that, as we did not encounter it. Could you please elaborate on the steps that were taken leading up to the expected error? It would be helpful to include where that error presented itself as well. Additionally, it would be useful to readers if the doc included more concrete steps that show exactly how and when you are specifying the raft config via the helm chart, as part of the above elaboration. Lastly, two small code changes. The first is minor, we use {
"title": "HA Cluster with TLS",
"path": "platform/k8s/helm/examples/ha-tls"
}, added in the file with the other HA cluster examples (under "HA Cluster with Raft" and above "HA Enterprise Cluster with Raft"), located here: vault/website/data/docs-nav-data.json Line 1295 in ff43365
|
Thanks for the review. I setup a completely new "kind" cluster (single node) to again understand our initial aim with this PR. We recognized there exists no guide that clearly shows how to setup a Vault cluster on Kubernetes with Raft integrated storage and TLS certificates. The primary purpose of this PR therefore was to address this gap in the documentation and provide guidance on how to setup a Vault Cluster with Helm, Raft integrated storage and TLS. Now, for the concrete instructions, this will obviously be a mix of the two guides mentioned in the beginning:
However, there are some changes needed to make it work in harmony (last but not least the concrete steps you mentioned). Following along both guides, I ended up with a mixed value file that looked like this: global:
enabled: true
tlsDisable: false
server:
injector:
enabled: false
ha:
enabled: true
raft:
enabled: true
config: |
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}
storage "raft" {
path = "/vault/data"
}
#storage "file" {
# path = "/vault/data"
#}
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
extraVolumes:
- type: secret
name: vault-server-tls # Matches the ${SECRET_NAME} from above
# Affinity Settings
# Commenting out or setting as empty the affinity variable, will allow
# deployment to single node services such as Minikube
# This should be either a multi-line string or YAML matching the PodSpec's affinity field.
affinity:
# standalone:
# enabled: true Did you end up with something similar? How I constructed this YAML:
One further note. I followed all the steps regarding the creation of the CSR and the signing of the I hope that you could understand how we came up with the above YAML config file as a basis to reconstruct the error which presented itself regarding the certificates. If not, we should make this very clear initially. Now, back to the error that we experienced about the TLS certificate only being signed for the k8s cluster internal names, not the Pod name. If you throw the above YAML at at k8s cluster, this should deploy just fine, also on single-node "kind" clusters: helm install vault hashicorp/vault -f custom-values.yaml I skipped the basic setup of Helm and the additional HashiCorp repo here (prerequisite). We can then go on to initialize and unseal the first cluster node kubectl exec -it vault-0 -- sh
/ $ vault operator init -key-shares=1 -key-threshold=1
/ $ vault operator unseal <UNSEAL_KEY_HERE> This works fine for the first node, because no communication to other nodes (where the TLS issue appears) is involved. So we go ahead and join the second Vault Pod, kubectl exec -it vault-1 -- sh
/ $ vault operator raft join https://vault-0.vault-internal:8200
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:
* failed to join raft cluster: failed to join any raft leader node This is were we hit our error, the main reason for opening this PR and were we were not quite clear how to proceed. The issue at hand will be visible in the log of the node we tried to join,
As you can see, by default, the join attempt was made with the Pod name "vault-0" which is not part of the certificate Common Name (CN) or Subject Alternate Name (SAN). This is an issue, because during bootstrapping of the Raft cluster, the first request is made to the HTTP API Port of the other node directly, and if the certificate names do not match the requested name this will fail w/ the error above. The general process is also nicely described here: https://www.vaultproject.io/docs/concepts/integrated-storage#vault-networking-recap This is also the part, were we would like to clarify the documentation, especially, regarding the configuration of the Basically, we found two ways to make the configuration work so that we can join the Vault Pods by their Pod names and still use the CSR and the suggested CN/SAN from the existing guide:
Both ways turned out to work just fine, with the first option ( But in any case, we found these two configuration options relevant, when attempting to create a cluster with Raft integrated storage and TLS on Kubernetes. Thanks for pointing out the details regarding the formatting ( |
This PR appears abandoned. If there is no activity on the PR by September 8, it will be closed without merging. |
Hi @schavis can we please catch up and not silently close this PR here? Last time I checked, I was waiting for some feedback from your side. I'm still convinced that this additional deployment example for "HA with Raft storage backend + TLS" in the docs for "Helm chart examples" would be very helpful deployment guidance for anyone trying to deploy a highly available HashiCorp Vault cluster on Kubernetes with TLS. In that sense, it's the lovechild of the "Standalone with TLS" and the "HA with Raft" example :) What do you think? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for drafting these examples :)
I made suggestions to keep things in line with our style guide and to also help improve the scannability of the document. Let me know if you have any questions or concerns.
Describes how to set up a Raft HA Vault cluster with TLS certificate | ||
--- | ||
|
||
# Raft HA Server with TLS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# Raft HA Server with TLS | |
# Raft HA server with TLS |
Style correction: use sentence case for headings
Follow the steps from the example [HA Vault Cluster with Integrated Storage](/docs/platform/k8s/helm/examples/ha-with-raft) to build the cluster. | ||
|
||
Follow the examples and instructions in [Standalone Server with TLS](/docs/platform/k8s/helm/examples/standalone-tls) to create a TLS certificate. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follow the steps from the example [HA Vault Cluster with Integrated Storage](/docs/platform/k8s/helm/examples/ha-with-raft) to build the cluster. | |
Follow the examples and instructions in [Standalone Server with TLS](/docs/platform/k8s/helm/examples/standalone-tls) to create a TLS certificate. | |
## Before you start | |
1. Follow the steps in [HA Vault Cluster with Integrated Storage](/docs/platform/k8s/helm/examples/ha-with-raft) to build your cluster. | |
1. Follow the steps in [Standalone Server with TLS](/docs/platform/k8s/helm/examples/standalone-tls) to create a TLS certificate. |
Suggest making it clear that these are (essentially) prerequisites to following the instructions you provide below.
Follow the steps from the example [HA Vault Cluster with Integrated Storage](/docs/platform/k8s/helm/examples/ha-with-raft) to build the cluster. | ||
|
||
Follow the examples and instructions in [Standalone Server with TLS](/docs/platform/k8s/helm/examples/standalone-tls) to create a TLS certificate. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
## Bootstrap your Raft cluster | |
Suggest adding a heading to make it obvious what the suggestions below will have folks do
|
||
Follow the examples and instructions in [Standalone Server with TLS](/docs/platform/k8s/helm/examples/standalone-tls) to create a TLS certificate. | ||
|
||
Before cluster initialization and without proper configuration, the following warning is to be expected: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before cluster initialization and without proper configuration, the following warning is to be expected: | |
Without proper configuration, you will see the following warning before cluster initialization: |
style correction: write in active voice
core: join attempt failed: error="error during raft bootstrap init call: Put "https://vault-${N}.${SERVICE}:8200/v1/sys/storage/raft/bootstrap/challenge": x509: certificate is valid for ${SERVICE}, ${SERVICE}.${NAMESPACE}, ${SERVICE}.${NAMESPACE}.svc, ${SERVICE}.${NAMESPACE}.svc.cluster.local, not vault-${N}.${SERVICE}" | ||
``` | ||
|
||
The concepts for [Integrated Storage and TLS](/docs/concepts/integrated-storage#integrated-storage-and-tls) elaborate three possible solutions to mitigate this TLS verification warning and bootstrap the Raft cluster, two of which are discussed in further detail here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The concepts for [Integrated Storage and TLS](/docs/concepts/integrated-storage#integrated-storage-and-tls) elaborate three possible solutions to mitigate this TLS verification warning and bootstrap the Raft cluster, two of which are discussed in further detail here. | |
The concepts overview for [Integrated Storage and TLS](/docs/concepts/integrated-storage#integrated-storage-and-tls) covers the various options for mitigating TLS verification warnings and bootstraping your Raft cluster. | |
The examples below demonstrate two specific solutions. Both solutions ensure that the common name (CN) used for the `leader_api_addr` in the Raft stanza matches the name(s) listed in the TLS certificate. |
I suggest moving this up to the beginning of the document (before the suggested "Before you start" section) as it's a useful introduction to what your document will cover. Also, I would add the last sentence from your doc here as well since it provides a good summary of what your specific examples do.
Style correction: write in active voice.
|
||
The concepts for [Integrated Storage and TLS](/docs/concepts/integrated-storage#integrated-storage-and-tls) elaborate three possible solutions to mitigate this TLS verification warning and bootstrap the Raft cluster, two of which are discussed in further detail here. | ||
|
||
The warning disappears when the the [expected TLS servername](/docs/concepts/integrated-storage#autojoin-with-tls-servername) is correctly configured using [`leader_tls_servername`](/docs/configuration/storage/raft#leader_tls_servername) in the Raft stanza (`${CN}` in the example below): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The warning disappears when the the [expected TLS servername](/docs/concepts/integrated-storage#autojoin-with-tls-servername) is correctly configured using [`leader_tls_servername`](/docs/configuration/storage/raft#leader_tls_servername) in the Raft stanza (`${CN}` in the example below): | |
## Solution 1: Use auto-join and set the TLS server in your Raft configuration | |
The join warning disappears if you use autojoin and set the expected TLS server name (`${CN}`) with [`leader_tls_servername`](/docs/configuration/storage/raft#leader_tls_servername) in the Raft stanza for your Vault configuration. | |
For example: |
Suggest making it easier for folks scanning the doc to differentiate between the two solutions you're demonstrating.
Style correction: write in active voice
|
||
The warning disappears when the the [expected TLS servername](/docs/concepts/integrated-storage#autojoin-with-tls-servername) is correctly configured using [`leader_tls_servername`](/docs/configuration/storage/raft#leader_tls_servername) in the Raft stanza (`${CN}` in the example below): | ||
|
||
```hcl |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be useful to highlight the important line for folks so they know which part of the Raft config you're talking about. You can highlight the relevant line by wrapping the CodeBlockConfig
component around your code example (with whitespace before/after each tag) and noting the specific line(s) you want to highlight. For example:
<CodeBlockConfig highlight="6,14,22">
YOUR_CODE_BLOCK
</CodeBlockConfig>
} | ||
``` | ||
|
||
Alternatively, specify one `retry_join` which references the [name of the load balancer](/docs/concepts/integrated-storage#load-balancer-instead-of-autojoin) instead: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively, specify one `retry_join` which references the [name of the load balancer](/docs/concepts/integrated-storage#load-balancer-instead-of-autojoin) instead: | |
## Solution 2: Add a load balancer to your Raft configuration | |
If you have a load balancer for your Vault cluster, you can add a single `retry_join` stanza to your Raft configuraiton and use the load balancer address for `leader_api_addr`. | |
For example: |
Same suggestion as above: make it easier on folks scanning quickly to find the start of your second recommended solution
``` | ||
|
||
Alternatively, specify one `retry_join` which references the [name of the load balancer](/docs/concepts/integrated-storage#load-balancer-instead-of-autojoin) instead: | ||
```hcl |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you want to highlight the relevant line for this example:
<CodeBlockConfig highlight="5">
YOUR_CODE_BLOCK
</CodeBlockConfig>
} | ||
``` | ||
|
||
Both configuration options ensure that the the common name (CN) used for the `leader_api_addr` in the Raft stanza matches the name(s) listed in the TLS certificate. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both configuration options ensure that the the common name (CN) used for the `leader_api_addr` in the Raft stanza matches the name(s) listed in the TLS certificate. |
Suggest moving this statement to earlier in the doc (I added it to a previous suggestion).
Would you mind closing this PR now the update is captured in PR #22714? |
We were having a hard time to find the connections/links to the Raft concepts and the discussion on that page related to TLS server name verification, especially during the initial bootstrap phase.
This request creates the links between the examples for Helm Chart usage and the concepts related to the server name verification.
Additionally, it includes the examples that we used to understand the relevance of matching server name in Raft stanza and actual TLS certificate.
It might prove helpful others as well.
Delimitation: No example is given for the third alternative, using a certificate with a CIDR range.
Co-authored-by: Pascal Reeb [email protected]