AWS CLIv2 and AWS SSO auth fails

[stevie-]

We are using new aws cli v2 SSO auth feature. This works with aws cli, but terragrunt fails to get credentials.

$ aws --version
aws-cli/2.0.6 Python/3.8.2 Darwin/18.7.0 botocore/2.0.0dev10

.aws/config

[profile 123456789012-AWSAdministratorAccess]
sso_start_url = https://d-123456789.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 123456789012
sso_role_name = AWSAdministratorAccess
region = eu-west-1
output = json

Check aws cli works

$ export AWS_PROFILE=123456789012-AWSAdministratorAccess
$ aws sso login
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.eu-west-1.amazonaws.com/

Then enter the code:

REDACTED
Successully logged into Start URL: https://d-123456789.awsapps.com/start
$ aws s3 ls
2020-04-08 08:09:44 tf-states.lab.REDACTED

$ aws sts get-caller-identity
{
    "UserId": "REDACTED:redacted@redacted",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_AWSAdministratorAccess_abcdef12345/redacted@redacted"
}
$ terragrunt plan --terragrunt-source ~/REDACTED/git/tf-modules//aws/bootstrap
[terragrunt] 2020/04/09 10:38:44 Terragrunt Version: v0.23.6
[terragrunt] 2020/04/09 10:38:44 Reading Terragrunt config file at /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/terragrunt.hcl
[terragrunt] 2020/04/09 10:38:44 Did not find any locals block: skipping evaluation.
[terragrunt] 2020/04/09 10:38:44 Running command: /Users/REDACTED/git/cloud-config/terraform/../scripts/aws/get_admin_roles.sh
AWSReservedSSO_AWSAdministratorAccess_abcdef12345[terragrunt] 2020/04/09 10:38:46 run_cmd output: [AWSReservedSSO_AWSAdministratorAccess_abcdef12345]
[terragrunt] 2020/04/09 10:38:46 Found locals block: evaluating the expressions.
[terragrunt] 2020/04/09 10:38:46 Evaluated 5 locals (remaining 8): cloud, environment, default_empty_yaml, api_domain, aws_vars
[terragrunt] 2020/04/09 10:38:46 Evaluated 3 locals (remaining 5): terraform_module_local_path, module_settings_path, aws_root_domain
[terragrunt] 2020/04/09 10:38:46 Evaluated 2 locals (remaining 3): terraform_module_path, terraform_module_version
[terragrunt] 2020/04/09 10:38:46 Evaluated 3 locals (remaining 0): terraform_module_name, default_tags, terraform_source_default
[terragrunt] [/Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap] 2020/04/09 10:38:51 Running command: terraform --version
[terragrunt] 2020/04/09 10:38:51 Downloading Terraform configurations from file:///Users/REDACTED/git/tf-modules into /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro
[terragrunt] 2020/04/09 10:38:51 Copying files from /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap into /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap
[terragrunt] 2020/04/09 10:38:51 Setting working directory to /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap
[terragrunt] 2020/04/09 10:38:51 The file path /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/provider.tf already exists and if_exists for code generation set to "overwrite". Regenerating file.
[terragrunt] 2020/04/09 10:38:51 Generated file /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/provider.tf.
[terragrunt] 2020/04/09 10:38:51 The file path /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.
[terragrunt] 2020/04/09 10:38:51 Generated file /Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap/.terragrunt-cache/bL_WKXycpKiko0WERZJHEmdsjyA/fnop9qC_WS7CKETpjnoJj_MM_ro/aws/bootstrap/backend.tf.
[terragrunt] [/Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap] 2020/04/09 10:38:51 Initializing remote state for the s3 backend
[terragrunt] 2020/04/09 10:38:51 Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[terragrunt] 2020/04/09 10:38:51 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[stevie-]

S3 bucket config from terragrunt.hcl. We checked setting AWS profile here as well. No change.

# Configure Terragrunt+Terraform to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"

  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }

  config = {
    bucket         = "tf-states.${local.environment}.${local.aws_root_domain}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-central-1"
    encrypt        = true
    dynamodb_table = "tf-states-lock.${local.environment}.${local.aws_root_domain}"

    s3_bucket_tags      = local.default_tags
    dynamodb_table_tags = local.default_tags
    # save costs - since we can not add lifecycle rules to clean up old versions
    skip_bucket_versioning = true
  }
}

[brikis98]

I believe the new SSO functionality only works with the newest version of the AWS SDK, so all CLI tools that use the SDK (such as Terragrunt) would need to be updated first. Would you be up for a PR to bump the AWS SDK version? You could test locally to see if that fixes the issue for you.

[stevie-]

PR = PullRequest -- my GOlang is very basic. I doubt that I can handle it.

Found this changelog:
https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md#release-v12530-2019-11-07

v1.25.30 added support for SSO, maybe related.

[stevie-]

I checked the following, but no success.

11:05 $ git diff
diff --git a/go.mod b/go.mod
index c958777..4f9ec28 100644
--- a/go.mod
+++ b/go.mod
@@ -4,7 +4,7 @@ go 1.13

 require (
        cloud.google.com/go/storage v1.6.0
-       github.com/aws/aws-sdk-go v1.29.34
+       github.com/aws/aws-sdk-go v1.30.7
        github.com/creack/pty v1.1.9
        github.com/fatih/color v1.9.0
        github.com/fatih/structs v1.1.0
diff --git a/go.sum b/go.sum
index 1042b8b..7426f99 100644
--- a/go.sum
+++ b/go.sum
@@ -99,6 +99,8 @@ github.com/aws/aws-sdk-go v1.25.3/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN
 github.com/aws/aws-sdk-go v1.27.1/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
 github.com/aws/aws-sdk-go v1.29.34 h1:yrzwfDaZFe9oT4AmQeNNunSQA7c0m2chz0B43+bJ1ok=
 github.com/aws/aws-sdk-go v1.29.34/go.mod h1:1KvfttTE3SPKMpo8g2c6jL3ZKfXtFvKscTgahTma5Xg=
+github.com/aws/aws-sdk-go v1.30.7 h1:IaXfqtioP6p9SFAnNfsqdNczbR5UNbYqvcZUSsCAdTY=
+github.com/aws/aws-sdk-go v1.30.7/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f/go.mod h1:AuiFmCCPBSrqvVMvuqFuk0qogytodnVFVSN5CeJB8Gc=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
 github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
@@ -330,6 +332,8 @@ github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANyt
 github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
+github.com/jmespath/go-jmespath v0.3.0 h1:OS12ieG61fsCg5+qLJ+SsW9NicxNkg3b25OyT2yCeUc=
+github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik=
 github.com/joefitzgerald/rainbow-reporter v0.1.0/go.mod h1:481CNgqmVHQZzdIbN52CupLJyoVwB10FQ/IQlF1pdL8=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/joyent/triton-go v0.0.0-20180313100802-d8f9c0314926/go.mod h1:U+RSyWxWd04xTqnuOQxnai7XGS2PrPY2cfGoDKtMHjA=

Build new version with go build and used this.
But same Error:

[terragrunt] [/Users/REDACTED/git/cloud-config/terraform/lab/aws/bootstrap] 2020/04/09 11:04:37 Initializing remote state for the s3 backend
[terragrunt] 2020/04/09 11:04:37 Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[terragrunt] 2020/04/09 11:04:37 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[lorengordon]

AWS SSO is going to be a mess across the community because it does not make credentials available in a way that works with the current credential chain. Each SDK will need to implement support for the new credential chain, and perhaps each tool also (depending on how they setup their chain).

For GO, see: aws/aws-sdk-go#3186

[jsaevitzon]

I also started getting these errors last night around 7pm.

[brikis98]

Ah, good catch @lorengordon... If this isn't implemented in the AWS Go SDK, then we won't be able to do anything until it is.

[lorengordon]

Fwiw, here is a workaround using credential_process until more SDKs support the AWS SSO credential chain...

https://github.com/benkehoe/aws-sso-credential-process

[urz9999]

In case like this one or other similar cases where AWS SSO result in incompatibilities with your library and you don't want to play with workarounds or complicated fixes, maybe you can give a try to our open-source project: https://github.com/Noovolari/leapp. It deals with AWS SSO authentication and accounts/roles retrieval then it creates short-lived temporary credentials in .aws/credentials to maximize compatibility with third party tools / sdks.

[hereandnow]

this should be fixed in the AWS Go SDK in version 1.37: https://github.com/aws/aws-sdk-go/releases/tag/v1.37.0

can we somehow get this into terragrunt? i am not that familiar with go, but for my understanding it should be only updating the version here:

terragrunt/go.mod

Line 7 in a7c0d43

github.com/aws/aws-sdk-go v1.29.34

?

can someone confirm? if so, i would be able to provide a PR

[idcmp]

I've checked out master and bumped aws-sdk-go to v1.37.6 and rebuilt.

I've got the following in ~/.aws/config:

[profile RoleNameHere-999999999999]
sso_start_url = https://my.awsapps.com/start
sso_region = us-east-1
sso_account_id = 999999999999
sso_role_name = RoleNameHere
region = us-east-1
output = json

I have AWS_PROFILE=RoleNameHere-999999999999 and can confirm running aws sts get-caller-identity works.

When I run my home-made terragrunt, I get:

ERRO[0006] /home/idcmp/src/stuff/terragrunt/terragrunt.hcl:4,46-65: Error in function call; Call to function "get_aws_account_id" failed: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors. 
ERRO[0006] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 

With both Terraform and the AWS Provider now supporting SSO credentials, it would be great if Terragrunt followed suit.

[brikis98]

Agreed! Is anyone interested in submitting a PR for this?

[z0mbix]

I can confirm that building terragrunt locally after updating go.mod to use go sdk version 1.37.7 (https://github.com/aws/aws-sdk-go/releases/tag/v1.37.7) that AWS SSO support now works.

[hereandnow]

@z0mbix : when you have already done it, can you provide a PR for this?

[z0mbix]

@z0mbix : when you have already done it, can you provide a PR for this?

Yup, will do.

[z0mbix]

I've created: #1537

[sylr]

I've created: #1537

I built it and can assert it fixes this issue for me. Thank you @z0mbix

EDIT: I might have been shouting victory too soon. With this PR terragrunt did not error before initializing the S3 backend to use for storing the tfstate & locks. I've not been further so far.

[ghost]

Weird that didn't work for me, but I'm glad it works in general. Thanks @z0mbix!

[z0mbix]

@kleidcmp Make sure you see my comments on the PR about credential_process, maybe that's your problem?

[z0mbix]

I've created: #1537

I built it and can assert it fixes this issue for me. Thank you @z0mbix

EDIT: I might have been shouting victory too soon. With this PR terragrunt did not error before initializing the S3 backend to use for storing the tfstate & locks. I've not been further so far.

Are you using the latest terraform aws provider version that is also required to support SSO:

hashicorp/terraform-provider-aws#10851 (comment)

[sylr]

Ok so I finally made it work by compiling both terraform and terragrunt linked against github.com/aws/[email protected].

My use case involves several AWS accounts spread in a tree looking like that:

terragrunt-workspace/aws/dev/terragrunt.hcl
terragrunt-workspace/aws/dev/eu-west-1/terragrunt.hcl backend.tf main.tf ... etc
terragrunt-workspace/aws/dev/eu-west-3/terragrunt.hcl backend.tf main.tf ... etc
terragrunt-workspace/aws/prod/terragrunt.hcl
terragrunt-workspace/aws/prod/eu-west-1/terragrunt.hcl backend.tf main.tf ... etc
terragrunt-workspace/aws/prod/eu-west-3/terragrunt.hcl backend.tf main.tf ... etc

With terragrunt-workspace/aws/<account>/terragrunt.hcl:

remote_state {
  backend = "s3"
  config  = {
    profile        = "<account>" # IMPORTANT
    bucket         = "<account>-terragrunt"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-west-3"
    encrypt        = true
    dynamodb_table = "<account>-terragrunt"
  }
}

terragrunt-workspace/aws/<account>/<region>/main.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }
}

provider "aws" {
  region                      = "<region>"
  profile                     = "<account>"
  skip_credentials_validation = true
}

terragrunt-workspace/aws/<account>/<region>/backend.tf:

terraform {
  backend "s3" {
    profile = "<account>"
  }
}

~/.aws/config:

[default]
region = eu-west-1
output = json
csm_enabled = true

[profile dev]
sso_start_url = https://d-xxxxxxxxx.awsapps.com/start
sso_region = eu-west-1
sso_account_id = yyyyyyyyyyyyy
sso_role_name = RoleName

[profile prod]
sso_start_url = https://d-xxxxxxxxx.awsapps.com/start
sso_region = eu-west-1
sso_account_id = yyyyyyyyyyyyy
sso_role_name = RoleName

<account> needs to be replace by the appropriate profile name.

With this I can run terragrunt run-all plan in the aws dir without needing to set any AWS_PROFILE env var. This however requires that I have valid sso session for all the SSO profiles so I made a quick aws sso login wrapper.

#!/usr/bin/env bash

for account_dir in aws/*/; do
    aws sso login --profile="$(basename $account_dir)"
done

[oscarmorasu]

Thanks for starting this work @z0mbix

Note Terraform already supports SSO creds started with version 0.14.6 (for the backend) and in version 3.26.0 for the Provider

I had to pin Terraform 0.14.5 + Provider 3.25.0 until Terragrunt also supports SSO. The reason is I am not using AWS Key/Secret Key, but I leverage the aws-sso-credential-process to bridge the gap.

Basically my profiles in the .aws/config file look like this:

[profile my-profile]
credential_process = aws-sso-credential-process --profile my-profile
sso_start_url = https://myorg.awsapps.com/start
sso_region = us-west-2
sso_account_id = 1234567890
sso_role_name = MyRole
region = us-west-2
output = json

Terraform 0.14.6+ blows up because both the credential_process and the sso config are present. If I remove the credential_process now Terragrunt blows up because it's unaware of the sso config and needs the AWS Key/Secret Key.

[UrosCvijan]

@oscarmorasu How did you manage to work with terragrunt with these specific versions? I fixed the versions as you said, added the profile.. Added the profile in the provider section and in terragrunt.hcl for the remote_state backend configuration. When I run terragrung plan for example I get
Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id: eb986117-f29c-4c87-a999-f6a9af5c2224
Or on a newly created project I get the error when S3 backend should be created:

terragrunt plan --terragrunt-source ../../../../terraform//transit-gateway/
Remote state S3 bucket xxxxxxxxxxx does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y
ERRO[0004] Create S3 bucket with retry xxxxxxxxxxx returned an error: InvalidClientTokenId: The security token included in the request is invalid.

Am I missing something?

[oscarmorasu]

@UrosCvijan, did you install aws-sso-credential-process?
Actually that tool is deprecated, the author replaced it with aws-sso-util

[UrosCvijan]

Wow, in the meantime I think I managed...Or I made who knows what with lot of credentials etc.. I will have to do it all over again with everything :) Now it looks it might passed, but will have to double check. Didn't see that it is deprecated. I will try the new one.

[UrosCvijan]

Ok, so just if anyone else will be using the workaround. I installed aws-sso-credential-process, created all the profiles using aws-configure-sso-profile for my accounts, added also sso_interactive_auth=true, not sure if it is necessary, included profile in my provider section, as well as in remote_state section for the backend creation. There is something strange happening when I run it the first time, it complains about the permissions and says that it cant create the S3 bucket, but it creates it (just ignores all the options), it is created without versioning and without encryption, but it is there, that one and the dynamo table. Then plan/apply works as normal. With this I can easily switch between accounts without having to login to new profile or to fetch the environment variables (AWS access/secret key and token).
Hoping terragrunt will support this natively cause that would be awesome.

[brikis98]

We just updated the AWS Go SDK version in Terragrunt, which, in theory, should allow AWS SSO / AWS CLI v2 to work with the new release: https://github.com/gruntwork-io/terragrunt/releases/tag/v0.28.9 (binaries should show up shortly). Please give it a shot!