AWS Profiles

[pixie79]

Hi,

How does terragrunt interact with different AWS account credentials? Is there an option to set the profile in the config or will it respect the AWS_DEFAULT_PROFILE variable to get the correct API keys for use.

Thanks

Mark

[josh-padnick]

Hi @pixie79, thanks for the question!

Since terragrunt is written in Go and uses the AWS SDK for Go, the AWS authentication is handled completely by the AWS SDK.

You can read about the specific authentication hierarchy used by the AWS SDK, which clarifies that terragrunt will first look for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. If those are undefined, it will use the shared credentials file, and I'm guessing it will also respect the AWS_DEFAULT_PROFILE env var, though this would ultimately be up to the AWS SDK.

If this answered your question, feel free to close this issue. Otherwise, I'll leave it open for a bit in case you have any follow-up questions.

[mmorianos]

I notice you don't support the use of temporary credentials which require a 3rd environment variable, AWS_SESSION_TOKEN. We don't use IAM:users or issue long term access keys, instead we built a service that pulls temporary credentials via STS to provide access. It looks like the SDK supports STS but not sure the level of effort to include that with terragrunt.

[brikis98]

@mmorianos We are using the default credentials providers from the AWS Go SDK. Following that SDK code, it uses the CredChain function, which includes an EnvProvider, which, at least in theory, includes the AWS_SESSION_TOKEN environment variable.

Did you try setting the three environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN) and find that it didn't work? If so, what error did you get?

[ryangardner]

What's interesting is that it does seem to use the SESSION_TOKEN_ID to create the table for me, but then fails to write an entry in the statefile with it.

I deleted the lock table and tried from scratch with those environment variables exported (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN):

[terragrunt] 2016/09/13 10:32:07 Remote state is already configured for backend s3
[terragrunt] 2016/09/13 10:32:07 Attempting to acquire lock for state file sherpa-services-lock in DynamoDB
[terragrunt] 2016/09/13 10:32:07 Lock table terragrunt_locks does not exist in DynamoDB. Will need to create it just this first time.
[terragrunt] 2016/09/13 10:32:07 Creating table terragrunt_locks in DynamoDB
[terragrunt] 2016/09/13 10:32:07 Table terragrunt_locks is not yet in active state. Will check again after 10s.
[terragrunt] 2016/09/13 10:32:17 Table terragrunt_locks is not yet in active state. Will check again after 10s.
[terragrunt] 2016/09/13 10:32:28 Success! Table terragrunt_locks is now in active state.
[terragrunt] 2016/09/13 10:32:28 Attempting to create lock item for state file sherpa-services-lock in DynamoDB table terragrunt_locks
[terragrunt] 2016/09/13 10:32:28 InvalidClientTokenId: The security token included in the request is invalid
    status code: 403, request id: e56e58a3-79be-11e6-a898-cf1ede838a3b

So, part of it works (creating the table) but part of it doesn't work (creating the lock item in the table)

[brikis98]

@ryangardner Hm, if some of the requests work, I think that means the auth is working. Perhaps "The security token included in the request is invalid" is just a poorly worded error message that actually means you lack the IAM permissions to write to that table?

[brikis98]

@ryangardner Ah, after a quick google search, I have an alternative theory: when creating a lock, the terragrunt code calls the IAM GetUser API. As I found from this comment in Terraform's issue tracker (which is relevant, since Terraform also uses the AWS Go SDK), if you are using the AWS APIs with dynamic credentials generated by GetSessionToken, those credentials cannot be used to make IAM calls unless you are using MFA.

Could you test this theory out? Use the AWS CLI to make some random API call (e.g. list the contents of an S3 bucket) and make sure it works with those env vars. Then, try this:

aws iam get-user

I'm guessing you'll get the same "security token included in the request is invalid" error.

[mmorianos]

yep, that's the problem.

aws iam get-user

An error occurred (InvalidClientTokenId) when calling the GetUser operation: The security token included in the request is invalid

[brikis98]

Could you try enabling MFA and seeing if that fixes the issue?

[mmorianos]

we experienced a similar problem with terraform and worked with them to modify the code and use the STS endpoint documented here... hashicorp/terraform#6536

[brikis98]

@mmorianos Ah, nice find! I think sts:GetCallerIdentity will work perfectly here.

Anyone interested in submitting a PR?

[ryangardner]

I submitted a PR for this - as a side bonus of switching to sts:getCallerIdentity you don't need to have the iam:getUser permission anymore

[brikis98]

A huge thanks to @ryangardner for submitting a fix. I've merged it in and just created a new release:

https://github.com/gruntwork-io/terragrunt/releases/tag/v0.0.11

The CircleCI build is running now and in 5-10 minutes, the new binaries should be available at the URL above. Please give them a shot and let me know if things are working for you!

[zerolaser]

so, The terragrunt will be retrieving aws credentials for remote state and dynamodb table. when terragrunt plan/apply is performed it in-turn runs the terraform plan/apply which again retrieves the aws credentials and applies the infrastructure. correct me if i am wrong.

[brikis98]

@zerolaser Yes, that sounds right. Is that a problem?

[zerolaser]

@brikis98 I do have multiple aws credentials set in ~/.aws/credentials. so when I pass profile = prod to the provider, The terraform is using prod credentials while terragrunt is using default credentials.Is there any work around for this ?

[brikis98]

@zerolaser Have you tried the AWS_DEFAULT_PROFILE environment variable?

[brikis98]

Or maybe it's just AWS_PROFILE? Not sure.

[zerolaser]

@brikis98 setting AWS_PROFILE worked. thanks.

[brikis98]

OK, I'm going to mark this bug as closed. If anyone is still having issues, let us know, and we can re-open.