Skip to content

Commit

Permalink
dr_replication: fix a race in the DR replication scenario
Browse files Browse the repository at this point in the history
Now that we are using a new module for secrets engines verification
without retries we could run into a race:

```
╷
│ Error: exit status 1
│
│ Error: Execution Error
│
│   with module.verify_replicated_data_during_failover.enos_remote_exec.auth_login_testuser,
│   on ../../modules/verify_secrets_engines/modules/read/auth.tf line 8, in resource "enos_remote_exec" "auth_login_testuser":
│    8: resource "enos_remote_exec" "auth_login_testuser" {
│
│ failed to execute commands due to: running script:
│ [/Users/ryan/code/hashi/vault/enos/modules/verify_secrets_engines/scripts/auth-userpass-login.sh]
│ failed, due to: 1 error occurred:
│       * executing script
│ auth/userpass/login/testuser: Error making API request.
│
│ URL: PUT http://[::1]:8200/v1/auth/userpass/login/testuser
│ Code: 503. Errors:
│
│ * Vault is sealed
```

We solve this by waiting for the cluster to be fully unsealed before
attempting to login during the data read.

Signed-off-by: Ryan Cragun <[email protected]>
  • Loading branch information
ryancragun committed Sep 9, 2024
1 parent e9ae8b8 commit 536a695
Showing 1 changed file with 30 additions and 2 deletions.
32 changes: 30 additions & 2 deletions enos/enos-scenario-dr-replication.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -990,11 +990,37 @@ scenario "dr_replication" {
}
}

step "verify_new_primary_cluster_unsealed" {
description = global.description.verify_vault_unsealed
module = module.vault_verify_unsealed
depends_on = [
step.wait_for_demoted_cluster_leader,
]

providers = {
enos = local.enos_provider[matrix.distro]
}

verifies = [
quality.vault_auto_unseals_after_autopilot_upgrade,
quality.vault_seal_awskms,
quality.vault_seal_pkcs11,
quality.vault_seal_shamir,
]

variables {
hosts = step.get_secondary_cluster_ips.follower_hosts
vault_addr = step.create_secondary_cluster.api_addr_localhost
vault_install_dir = global.vault_install_dir[matrix.artifact_type]
}
}

step "verify_replicated_data_during_failover" {
description = global.description.verify_secrets_engines_read
module = module.vault_verify_secrets_engines_read
depends_on = [
step.wait_for_demoted_cluster_leader
step.wait_for_demoted_cluster_leader,
step.verify_new_primary_cluster_unsealed,
]

providers = {
Expand Down Expand Up @@ -1025,7 +1051,9 @@ scenario "dr_replication" {
so that secondary clusters can utilize it.
EOF
module = module.generate_secondary_public_key
depends_on = [step.verify_replicated_data_during_failover]
depends_on = [
step.verify_replicated_data_during_failover,
]

verifies = quality.vault_api_sys_replication_dr_primary_secondary_token_write

Expand Down

0 comments on commit 536a695

Please sign in to comment.