Vault no longer respects AWS_ROLE_ARN or AWS_WEB_IDENTITY_TOKEN_FILE for AWS KMS

[dqsully]

Describe the bug
AWS KMS seals no longer respect the AWS_ROLE_ARN or AWS_WEB_IDENTITY_TOKEN_FILE environment variables, which are required for assuming IAM roles via Kubernetes ServiceAccount tokens. Instead, Vault attempts to use the EC2 instance's IAM role (if available) to access the KMS key instead of using the Kubernetes ServiceAccount.

To Reproduce
Steps to reproduce the behavior:

1. Create an AWS EKS cluster
2. Connect the EKS cluster as an OIDC provider for AWS IAM
   
3. Create an AWS KMS key
4. Create an AWS IAM role with an assume-role policy authorizing that OIDC provider, and an inline policy authorizing that IAM role for actions on the KMS key you created
5. Deploy Vault with an "awskms" seal, setting only kms_key_id, and adding a ServiceAccount annotation eks.amazonaws.com/role-arn: <IAM role ARN>

Expected behavior
Vault should assume the IAM role configured in the Kubernetes ServiceAccount annotation and referenced by AWS_ROLE_ARN (injected by EKS because of the annotation), using the Kubernetes ServiceAccount token file referenced by AWS_WEB_IDENTITY_TOKEN_FILE (also injected by EKS) for authentication with AWS.

Environment:

• Vault Server Version: 1.14.0
• Server Operating System/Architecture: running on EKS 1.24, deployed using Vault's official Helm chart v0.25.0

Vault server configuration file(s):

ui = true
listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  x_forwarded_for_authorized_addrs = ["####", "####"]
  x_forwarded_for_reject_not_present = false
}
storage "consul" {
  path = "vault"
  address = "HOST_IP:8500"
}

service_registration "kubernetes" {}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "######"
}

Additional context
There is an easy workaround for this bug, which is to set role_arn and web_identity_token_file in the seal settings like so:

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "######"

  # Fixes for Vault 1.14+
  role_arn = "arn:aws:iam::<account ID>:role/<role name>"
  web_identity_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
}

Also, as far as I could trace it, the issue seems to come from this list of approved(?) environment variables for the AWS KMS wrapper?: 254d8f8#diff-8669cb5f3518deb7d1841c405e7e8b222348751cf85f81e6077a1184e9ed767dR15-R23

Hopefully the fix is as easy as adding AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN to that list.

[ubajze]

We are running into a similar issue, but unfortunatelly the workaround is not working for us.

I still get:

vault Error parsing Seal configuration: error fetching AWS KMS wrapping key information: NoCredentialProviders: no valid providers in chain. Deprecated.
vault     For verbose messaging see aws.Config.CredentialsChainVerboseErrors

This is how our seal configuration looks like:

    # Auto unseal is enabled using the AWS KMS service
    seal "awskms" {
      region = "us-east-2"
      kms_key_id = "KMS_KEY_ID"
      web_identity_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
    }

[dqsully]

@ubajze I apologize, I was in a bit of a rush to solve this issue on our own side, and I missed that apparently the AWS_ROLE_ARN environment variable is also not being recognized, so you have to set the role_arn setting in the seal too. I did that on my own end just in case, but I assumed that it wasn't important. I'll update my original comment with that additional info.

[ubajze]

@dqsully it is working after adding the role_arn. Thank you for the update.

[dqsully]

@hsimon-hashicorp I don't know the conventions here but would this be better labeled under core/seal since that's the component that's affected? It's unrelated to the AWS secrets backend or the OIDC auth method, and while my example runs on Kubernetes with the Helm chart, this issue isn't unique to Kubernetes.

[igor-nikiforov]

We have the same issue after upgrade to 1.14.0. This config no longer works:

seal "awskms" {
    region     = "${data.aws_region.current.name}"
    kms_key_id = "${module.aws_kms_key.key_id}"
}

[tolleiv]

This seems to be also relevant for the s3 storage backend which doesn't support the web_identity_token_file and role_arn configuration.

[shivshav]

Same issue here. We're in the middle of trying to get IRSA incorporated into our environments when we upgraded our staging instance to 1.14.0 to account for this CVE

Looks like this may be related to this issue about the same thing with the storage configuration and potentially this issue in the go-secure-stdlib library?

[igoritos22]

We have the same situation here.
With same EKS and CHart versions.

[rissson]

This also breaks the AWS secret backend.

[heatherezell]

Hi folks, thanks for the issue report, the repro steps, and the comments about this problem! Our engineering teams are working on a fix to be included in an upcoming release. Thanks for your patience in the meantime!

[fairclothjm]

Resolved by #21951.

The fix will be available in Vault v1.14.1

[Suman-Mohan]

Hi ,
We are still seeing similar issues in Vault v1.14.1 . Is anyone else still facing this issue?

[nitrogear]

I still can reproduce the issue. Used the latest helm chart v0.25.0 with Vault v1.14.0 and v1.14.1.
Nothing below helped:

• I set env var AWS_ROLE_ARN
• added annotation to SA vault with "eks.amazonaws.com/role-arn"
• added role_arn/web_identity_token_file to CM
  Vault still uses IAM role assigned to EC2 instance, not the one I created for it
  Here is the error:

Error parsing Seal configuration: error fetching AWS KMS wrapping key information: AccessDeniedException: User: arn:aws:sts::XXX:assumed-role/qa-eks-worker-nodes-NodeInstanceRole-SBBNZTGPBP3D/i-0b1b5e3943a6479a7 is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-central-1:XXX:key/31a61ccb-71f4-44a8-abab-c78487d43998 because no identity-based policy allows the kms:DescribeKey action
	status code: 400, request id: 10061045-418a-4639-9885-64b991c7c890

[nitrogear]

I got it working with Vault v1.14.1
The issue was in SA annotation. If you don't add the annotation "eks.amazonaws.com/role-arn" to the Vault's SA account then EKS won't mount token "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" to vault pod.
Without that Vault ignores role_arn/web_identity_token_file in the configuration file