v1.14.0 breaks AWS EKS IRSA credentials for DynamoDB backend

[SeanKnight]

Describe the bug

Startup logs:

2023-06-27T00:36:30.759Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
Error initializing storage of type dynamodb: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

To Reproduce

Steps to reproduce the behavior:

1. Install vault on EKS cluster via helm chart (0.23.0 or 0.25.0) using image tag 1.14.0

Expected behavior
Vault starts up like it did on version 1.13.4

Environment:

• Vault Server Version (retrieve with vault status): 1.14.0
• Vault CLI Version (retrieve with vault version): n/a
• Server Operating System/Architecture: linux/amd64

Vault server configuration excerpt:

storage "dynamodb" {
  ha_enabled = "true"
  region = "us-west-2"
  table = "vault"
}

Helm values excerpt:

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.14.0"
  serviceAccount:
    annotations:
      "eks.amazonaws.com/role-arn": MY_ROLE_ARN

Additional context
Image tag 1.13.4 works but 1.14.0 does not

Possibly related since the version of hashicorp/go-secure-stdlib has changed in go.mod for vault 1.14.0:

• Following v0.2.1, applications using Vault cannot login via AWS Method on EKS go-secure-stdlib#71

[pjain-fastly]

We are running into the same issue and would appreciate a resolution. The new support for ACME is a feature we have been waiting for, since a long time.

[ubajze]

We are running into the similar issue after upgrading from 1.13 to 1.14, but in our case the seal configuration is causing problems:

vault Error parsing Seal configuration: error fetching AWS KMS wrapping key information: NoCredentialProviders: no valid providers in chain. Deprecated.
vault     For verbose messaging see aws.Config.CredentialsChainVerboseErrors

The issue #21478 mentions a workaround, but it is not working for us.

[ubajze]

@SeanKnight Based on #21478 you may need to add role_arn and web_identity_token_file to your configuration block.

[dave4086]

I added both role_arn and web_identity_token_file to the storage config block (example of config below) and am still receiving role assumption errors. Vault is trying to use the worker node role rather than the role that is passed in through the role_arn setting.

storage "dynamodb" {
    ha_enabled = "true"
    table = "vault-dynamo-table"
    region = "us-east-1"
    role_arn = "arn:aws:iam::012345678901:role/nonprod-vault-role"
    web_identity_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
 }

[SeanKnight]

@ubajze adding role_arn and web_identity_token_file did not work. I'm not even getting to the unseal stage because storage is not loading.

[DeepakRai94]

We have exactly the same issue,

I tried to add, role_arn and web_identity_token_file to the both storage config block and awskms block but still getting role assumption errors.

Error initializing storage of type dynamodb: AccessDeniedException: User: arn:aws:sts:::assumed-role/node-role is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:eu-west-1::table/vault because no identity-based policy allows the dynamodb:DescribeTable action

storage "dynamodb" {
    ha_enabled = "true"
    table = "vault-dynamo-table"
    region = "eu-east-1"
    # Fixes for Vault 1.14+
    role_arn = "arn:aws:iam::<account ID>:role/<role name>"
    web_identity_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
 }

seal "awskms" {
  region     = "eu-east-1"
  kms_key_id = "######"

  # Fixes for Vault 1.14+
  role_arn = "arn:aws:iam::<account ID>:role/<role name>"
  web_identity_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
}

[ddanijel]

Getting exactly the same error. Vault pods are ignoring the IRSA and using the node role. We have to use the IRSA because of multi-tenancy. Hope you can prioritize this issue.

[mtavaresmedeiros]

same thing here, the vault pods are ignoring the IRSA.
Vault 1.14.0

[jbouse]

I'm seeing this error as well and find it odd that the suggested fixes are to include the role_arn and web_identity_token_file settings directly to this seems to indicate that with the upgrade to 1.14.0 Vault has decided to no longer look for and/or acknowledge the env variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE that are recognized by the AWS SDK.

[fairclothjm]

Hello and thanks for reporting. We will be looking into this issue soon. I will post any updates here as soon as I get them.

[fulcrum29]

Ran into the same issue. Any resolution in sight?

[bdwyertech]

The issue is because of how awsutil is used to derive credentials. The NewCredentialsConfig constructor method needs to be called instead of starting with an empty struct. the logic to determine which method to use to derive credentials has been moved from GenerateCredentialChain to NewCredentialsConfig.

1. The go-kms-wrapping library needs to be fixed to fix the KMS auto unseal.
   hashicorp/go-kms-wrapping@0b76b18

2. Vault itself needs to be updated to call NewCredentialsConfig instead of sucking in an empty struct.

See #21930