Vault v0.10.0 with backend etcd v3 Problem on HA #4561
Comments
|
How are you stopping it? |
|
i tried two ways, to reboot the machine using linode dashboard, and using docker
|
|
If you are stopping Vault, it can't redirect requests, because Vault isn't running. Requests get forwarded to the active node when Vault is unsealed in standby mode. |
victorr
pushed a commit
that referenced
this issue
Aug 24, 2023
* Run buf format. * Add buf.lock to ensure go-kms-wrapping module is imported.
victorr
added a commit
that referenced
this issue
Aug 24, 2023
* Seal HA: Use new SealWrappedValue type to abstract seal wrapped values Introduce SealWrappedValue to abstract seal wrapped values. Make SealWrappedValue capable of marshalling into a BlobInfo, when there is plaintext or a single encryption, or to a custom serialization consisting of a header, length and a marshalled MultiWrapValue protobuf. * Vault-13769: Support configuring and using multiple seals for unsealing * Make sealWrapBackend start using multiple seals * Make seal.Access no longer implement wrapping.Wrapper. Instead, add the Encrypt and Decrypt methods to the Access interface. * Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue. Move the snapshot.Sealer implementation to the vault package to avoid circular imports. * Update sealWrapBackend to use multiple seals for encryption. Use all the encryption wrappers when storing seal wrapped values. Try do decrypt using the highest priority wrapper, but try all combinations of encrypted values and wrappers if necessary. * Allow the use of multiple seals for entropy augmentation Add seal_name variable in entropy stanza Add new MultiSourcer to accommodate the new entropy augmentation behavior. * Individually health check each wrapper, and add a sys/seal-backend-status endpoint. * Address a race, and also a failed test mock that I didn't catch * Track partial wrapping failures... ... where one or more but not all access.Encrypts fail for a given write. Note these failures by adding a time ordered UUID storage entry containing the path in a special subdirectory of root storage. Adds a callback pattern to accomplish this, with certain high value writes like initial barrier key storage not allowing a partial failure. The followup work would be to detect return to health and iterate through these storage entries, rewrapping. * Add new data structure to track seal config generation (#4492) * Add new data structure to track seal config generation * Remove import cycle * Fix undefined variable errors * update comment * Update setSeal response * Fix setSealResponse in operator_diagnose * Scope the wrapper health check locks individually (#4491) * Refactor setSeal function in server.go. (#4505) Refactor setSeal function in server.go. * Decouple CreateSecureRandomReaderFunc from seal package. Instead of using a list of seal.SealInfo structs, make CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This brakes the denpency of package configutil on the seal package. * Move SealGenerationInfo tracking to the seal Access. * Move SealGenerationInfo tracking to the seal Access. The SealGenerationInfo is now kept track by a Seal's Access instead of by the Config object. The access implementation now records the correct generation number on seal wrapped values. * Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true. * Add MultiWrapValue protobuf message MultiWrapValue can be used to keep track of different encryptions of a value. --------- Co-authored-by: Victor Rodriguez <[email protected]> * Use generation to determine if a seal wrapped value is up-to-date. (#4542) * Add logging to seal Access implementation. * Seal HA buf format run (#4561) * Run buf format. * Add buf.lock to ensure go-kms-wrapping module is imported. * Vault-18958: Add unit tests for config checks * Add safety logic for seal configuration changes * Revert "Add safety logic for seal configuration changes" This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90. * changes and tests for checking seal config * add ent tests * remove check for empty name and add type into test cases * add error message for empty name * fix no seals test --------- Co-authored-by: divyapola5 <[email protected]> * Handle migrations between single-wrapper and multi-wrapper autoSeals * Extract method SetPhysicalSealConfig. * Extract function physicalSealConfig. The extracted function is the only code now reading SealConfig entries from storage. * Extract function setPhysicalSealConfig. The extracted function is the only code now writing SealConfig entries from storage (except for migration from the old recovery config path). * Move SealConfig to new file vault/seal_config.go. * Add SealConfigType quasy-enumeration. SealConfigType is to serve as the typed values for field SealConfig.Type. * Rename Seal.RecoveryType to RecoverySealConfigType. Make RecoverySealConfigType return a SealConfigType instead of a string. * Rename Seal.BarrierType to BarrierSealConfigType. Make BarrierSealConfigType return a SealConfigType. Remove seal.SealType (really a two-step rename to SealConfigType). * Add Seal methods ClearBarrierConfig and ClearRecoveryConfig. * Handle autoseal <-> multiseal migrations. While going between single-wrapper and multiple-wrapper autoseals are not migrations that require an unwrap seal (such as going from shamir to autoseal), the stored "barrier" SealConfig needs to be updated in these cases. Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have more than one wrapper; on the other hand, for autoseals with a single wrapper, SealConfig.Type is the type of the wrapper. * Remove error return value from NewAutoSeal constructor. * Automatically rewrap partially seal wrapped values on an interval * Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status. * Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call. * Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it. * Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code. * Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field. * Nil check the seal rewrap success callback before calling it. * Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance. * Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call. * Update partial rewrap check interval from 10 seconds to 1 minute. * Update a reference to SealGenerationInfo Rewrapped field to use new getter method. * Fix up some data raciness in partial rewrapping. * Account for possibly nil storage entry when retrieving partially wrapped value. * Allow multi-wrapper autoSeals to include disabled seal wrappers. * Restore propagation of wrapper configuration errors by setSeal. Function setSeal is meant to propagate non KeyNotFound errors returned by calls to configutil.ConfigureWrapper. * Remove unused Access methods SetConfig and Type. * Allow multi-wrapper autoSeals to include disabled seal wrappers. Make it possible for an autoSeal that uses multiple wrappers to include disabled wrappers that can be used to decrypt entries, but are skipped for encryption. e an unwrapSeal when there are disabled seals. * Fix bug with not providing name (#4580) * add suffix to name defaults * add comment * only change name for disabled seal * Only attempt to rewrap partial values when all seals are healthy. * Only attempt to rewrap partial values when all seals are healthy. * Change logging level from info to debug for notice about rewrap skipping based on seal health. * Remove stale TODOs and commented out code. --------- Co-authored-by: rculpepper <[email protected]> Co-authored-by: Larroyo <[email protected]> Co-authored-by: Scott G. Miller <[email protected]> Co-authored-by: Divya Pola <[email protected]> Co-authored-by: Matt Schultz <[email protected]> Co-authored-by: divyapola5 <[email protected]> Co-authored-by: Rachel Culpepper <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I had a problem on high availability on Vaul v0.10.0 with backend etcd v3.
the problem is i have two nodes in HA mode, when i stop the master node and send a write request to it, it responds with:
this is my vault config hcl file
`storage "etcd" {
address = "http://$ETCD_NODE_1:2379,http://$ETCD_NODE_2:2379"
etcd_api = "v3"
path = "vault/"
username = "$USERNAME"
password = "$PASSWORD"
ha_enabled = "true"
}
listener "tcp" {
address = "$VAULT_NODE_1:8200"
cluster_address = "$VAULT_NODE_1:8201"
tls_disable = 1
}
disable_mlock= true
api_addr = "http://$VAULT_NODE_1:8200"
cluster_addr = "https://$VAULT_NODE_1:8201"
`
and this is first node logs
2018-05-13T10:08:06.012Z [INFO ] core: vault is unsealed 2018-05-13T10:08:06.013Z [INFO ] core: entering standby mode 2018-05-13T10:30:08.635Z [INFO ] core: acquired lock, enabling active operation 2018-05-13T10:30:08.757Z [INFO ] core: post-unseal setup starting 2018-05-13T10:30:08.768Z [INFO ] core: loaded wrapping token key 2018-05-13T10:30:08.768Z [INFO ] core: successfully setup plugin catalog: plugin-directory= 2018-05-13T10:30:08.783Z [INFO ] core: successfully mounted backend: type=kv path=secret/ 2018-05-13T10:30:08.784Z [INFO ] core: successfully mounted backend: type=system path=sys/ 2018-05-13T10:30:08.784Z [INFO ] core: successfully mounted backend: type=identity path=identity/ 2018-05-13T10:30:08.784Z [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2018-05-13T10:30:08.858Z [INFO ] core: restoring leases 2018-05-13T10:30:08.858Z [INFO ] rollback: starting rollback manager 2018-05-13T10:30:08.864Z [INFO ] expiration: lease restore complete 2018-05-13T10:30:08.884Z [INFO ] identity: entities restored 2018-05-13T10:30:08.895Z [INFO ] identity: groups restored 2018-05-13T10:30:08.906Z [INFO ] core: post-unseal setup complete 2018-05-13T10:30:08.906Z [INFO ] core: core/startClusterListener: starting listener: listener_address=$VAULT_NODE_1:8201 2018-05-13T10:30:08.906Z [INFO ] core: core/startClusterListener: serving cluster requests: cluster_listen_address=VAULT_NODE_1:8201and this second node logs
2018-05-13T11:33:04.932Z [INFO ] core: vault is unsealed 2018-05-13T11:33:04.933Z [INFO ] core: entering standby modei also tried to stop the stand by node and send a write request and get the same error.
What i need is when i stop one node all the requests to this node redirected to the other nodes. How this can be done?
The text was updated successfully, but these errors were encountered: