Automate Getting Security Credentials from AWS_WEB_IDENTITY_TOKEN_FILE

[shreyasmm]

This feature request is related to EKS, which now supports IAM Roles for Service Accounts announced here.

When we assign a serviceAccount to a Pod or Deployment, In the pod environment we will have two environment variables

AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxxx:role/s3-access

When we run

# aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".

aws cli cannot directly create a session from AWS_WEB_IDENTITY_TOKEN_FILE environment variable automatically, Instead we need to run

# aws sts assume-role-with-web-identity \
 --role-arn $AWS_ROLE_ARN \
 --role-session-name mysession \
 --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
 --duration-seconds 1000 > /tmp/irp-cred.txt

# cat /tmp/irp-cred.txt
{
    "AssumedRoleUser": {
        "AssumedRoleId": "xxxxxxxxxxxxx:mysession", 
        "Arn": "arn:aws:sts::xxxxxxxxxxx:assumed-role/s3-access/mysession"
    }, 
    "Audience": "sts.amazonaws.com", 
    "Provider": "arn:aws:iam::xxxxxxxxxxxxx:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxx", 
    "SubjectFromWebIdentityToken": "system:serviceaccount:default:s3-sa", 
    "Credentials": {
        "SecretAccessKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", 
        "SessionToken": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", 
        "Expiration": "2019-09-10T09:59:06Z", 
        "AccessKeyId": "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
}

set the environment variable so aws-cli can work

# export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")"
# export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")"
# export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")"

When we run aws cli

# aws s3 ls
bucket-xxx
bucket-xxx
bucket-xxx

Can this whole process of creating a session from AWS_WEB_IDENTITY_TOKEN_FILE, getting credentials be automated in aws-cli could be helpful,
it finds diffcult to run the set of commands in each pods

Ref
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
aws/containers-roadmap#23

aws --version
aws-cli/1.16.102 Python/2.7.16 Linux/4.14.128-112.105.amzn2.x86_64 botocore/1.12.92

[realrill]

I struggle with similar issue on Vault-helm.
I get the expected two environment parameters (AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN), but my pod tries to authenticate with the node IAM.

I expected that the pod gets the IAM assigned.

Is it a bug or the application needs to deal with the further authentication against AWS on its own?

[micahhausler]

@shreyasmm Your AWS CLI is out of date. Please upgrade to 1.16.232

@realrill As far as I am aware Vault, does not yet support IAM for Service Accounts

[realrill]

@micahhausler If I understood correctly from your referred comment.
Does it mean I can not use init container with eg. awscli and populate env params from the session object becaue AssumeRoleWithWebIdentit not capable to call sts for this?

[micahhausler]

@realrill You can use the AWS cli, you just need to use a supported AWS CLI and the proper annotation on your Pod's Service Account. The EKS docs have a full walkthrough

[shreyasmm]

@micahhausler thank you for pointing out need to install latest awscli version. i have tested it its working fine in latest release

[micahhausler]

@shreyasmm Can this be closed out?

[shreyasmm]

@micahhausler yes sure

[bmbferreira]

/reopen

This issue is happening again, exactly as described originally by @shreyasmm for CLI version 2, more specifically:

> aws --version
aws-cli/2.2.29 Python/3.8.8 Linux/5.8.0-1038-aws exe/x86_64.ubuntu.20 prompt/off

I just rolled back to version 1.16.312 and it works.

[bmbferreira]

@micahhausler should I create a new issue or should this be reopen? thanks

[bmbferreira]

nevermind, the problem was that we were using a STS endpoint on our VPC and the security group was not added to it. It works on aws cli v1 because it doesn't use regional endpoints by default.

This issue can stay closed.