remote_state backend S3 bucket parameters not set when using an assumed role profile

[ryno75]

Overview

When using an AWS Profile configured to assume an IAM Role to run terragrunt, the bucket is created successfully but none of the bucket parameters are set. This includes properties like Taggin, SSE, Versioning, and Bucket Policy.

NOTE: Initially I thought this issue was isolated to AWS SSO Profiles but I have been able to reproduce this issue with any assumed role AWS Profile!

The following error is observed during the bucket creation/init sequence...

 Create S3 bucket with retry {REDACTED_BUCKET_NAME} returned an error: NoCredentialProviders: no valid providers in chain. Deprecated.

• Terragrunt Version: 0.28.22
• Terraform Version: 0.14.10

Steps to Reproduce

Testing with standard AWS IAM Role Assumption Profile

Show Identity

NOTE: The behavior is the same whether AWS_PROFILE and/or AWS_DEFAULT_PROFILE are set to the desired profile

❯ echo $AWS_PROFILE
terragrunt_remote_state_s3_test

❯ aws sts get-caller-identity
{
    "UserId": "AROASUTQMXEREDACTED00:botocore-session-1619019040",
    "Account": "{REDACTED}",
    "Arn": "arn:aws:sts::{REDACTED:assumed-role/CloudAdmin/botocore-session-1619019040"
}

Terragrunt.hcl

terraform {
  source = ".//test_module"
}

remote_state {
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  backend = "s3"
  config = {
    encrypt = true
    bucket  = "terragrunt-backend-s3-test003"
    key     = "terraform.tfstate"
    region  = "us-west-2"

    s3_bucket_tags = {
        Owner       = "foo"
        Environment = "test"
    }
  }
}

test_module/test.tf

# Does nothing but generate an output based on caller identity
provider "aws" {
  region = "us-west-2"
}

data "aws_caller_identity" "test" {}

output "caller_id" {
  value = data.aws_caller_identity.test
}

Terragrunt Apply (first-run with backend S3 bucket creation)

❯ terragrunt apply --terragrunt-log-level debug
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Running command: terraform --version          prefix=[/tmp/tgtest]
DEBU[0001] Terraform version: 0.14.10
DEBU[0001] Reading Terragrunt config file at /tmp/tgtest/terragrunt.hcl
DEBU[0001] Did not find any locals block: skipping evaluation.
DEBU[0001] Downloading Terraform configurations from file:///tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY
DEBU[0001] Copying files from /tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module
DEBU[0001] Setting working directory to /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module
DEBU[0001] Generated file /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module/backend.tf.  prefix=[/tmp/tgtest]
DEBU[0001] Initializing remote state for the s3 backend  prefix=[/tmp/tgtest]
Remote state S3 bucket terragrunt-backend-s3-test003 does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y
DEBU[0007] Create S3 bucket with retry terragrunt-backend-s3-test003  prefix=[/tmp/tgtest]
DEBU[0007] Creating S3 bucket terragrunt-backend-s3-test003  prefix=[/tmp/tgtest]
DEBU[0008] Created S3 bucket terragrunt-backend-s3-test003  prefix=[/tmp/tgtest]
DEBU[0008] Waiting for bucket terragrunt-backend-s3-test003 to be created  prefix=[/tmp/tgtest]
DEBU[0008] S3 bucket terragrunt-backend-s3-test003 created.  prefix=[/tmp/tgtest]
DEBU[0008] Enabling root access to S3 bucket terragrunt-backend-s3-test003  prefix=[/tmp/tgtest]
ERRO[0013] Create S3 bucket with retry terragrunt-backend-s3-test003 returned an error: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors. Sleeping for 10s and will try again.  prefix=[/tmp/tgtest]
DEBU[0023] Create S3 bucket with retry terragrunt-backend-s3-test003  prefix=[/tmp/tgtest]
DEBU[0023] Creating S3 bucket terragrunt-backend-s3-test003  prefix=[/tmp/tgtest]
DEBU[0023] Looks like you're already creating bucket terragrunt-backend-s3-test003 at the same time. Will not attempt to create it again.  prefix=[/tmp/tgtest]
DEBU[0023] Waiting for bucket terragrunt-backend-s3-test003 to be created  prefix=[/tmp/tgtest]
DEBU[0024] S3 bucket terragrunt-backend-s3-test003 created.  prefix=[/tmp/tgtest]
WARN[0024] Versioning is not enabled for the remote state S3 bucket terragrunt-backend-s3-test003. We recommend enabling versioning so that you can roll back to previous versions of your Terraform state in case of error.  prefix=[/tmp/tgtest]
DEBU[0024] Running command: terraform init               prefix=[/tmp/tgtest]

❯ cat terragrunt-debug.tfvars.json
{}%

❯ aws s3api get-bucket-tagging --bucket terragrunt-backend-s3-test003

An error occurred (NoSuchTagSet) when calling the GetBucketTagging operation: The TagSet does not exist

❯ aws s3api get-bucket-versioning --bucket terragrunt-backend-s3-test003

❯ aws s3api get-bucket-encryption --bucket terragrunt-backend-s3-test003

An error occurred (ServerSideEncryptionConfigurationNotFoundError) when calling the GetBucketEncryption operation: The server side encryption configuration was not found

❯ aws s3api get-bucket-policy --bucket terragrunt-backend-s3-test003

An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist

Testing with AWS SSO Profile

Terragrunt.hcl

terraform {
  source = ".//test_module"
}

remote_state {
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  backend = "s3"
  config = {
    encrypt = true
    bucket  = "terragrunt-backend-s3-test000"
    key     = "terraform.tfstate"
    region  = "us-west-2"
  }
}

test_module/test.tf

# Does nothing but generate an output based on caller identity
provider "aws" {
  region = "us-west-2"
}

data "aws_caller_identity" "test" {}

output "caller_id" {
  value = data.aws_caller_identity.test
}

Terragrunt Apply (first-run with backend S3 bucket creation)

❯ terragrunt apply --terragrunt-debug --terragrunt-log-level debug
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Evaluated 2 locals (remaining 0): region, env
DEBU[0000] Running command: terraform --version          prefix=[/tmp/tgtest]
DEBU[0001] Terraform version: 0.14.10
DEBU[0001] Reading Terragrunt config file at /tmp/tgtest/terragrunt.hcl
DEBU[0001] Found locals block: evaluating the expressions.
DEBU[0001] Evaluated 2 locals (remaining 0): region, env
DEBU[0001] Downloading Terraform configurations from file:///tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY
DEBU[0001] Copying files from /tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/mod
DEBU[0001] Setting working directory to /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/mod
DEBU[0001] The file path /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/mod/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.  prefix=[/tmp/tgtest]
DEBU[0001] Generated file /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/mod/backend.tf.  prefix=[/tmp/tgtest]
INFO[0001] Debug mode requested: generating debug file terragrunt-debug.tfvars.json in working dir /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/mod  prefix=[/tmp/tgtest]
DEBU[0001] The following variables were detected in the terraform module:  prefix=[/tmp/tgtest]
DEBU[0001] []                                            prefix=[/tmp/tgtest]
DEBU[0001] Variables passed to terraform are located in "/tmp/tgtest/terragrunt-debug.tfvars.json"  prefix=[/tmp/tgtest]
DEBU[0001] Run this command to replicate how terraform was invoked:  prefix=[/tmp/tgtest]
DEBU[0001] 	terraform apply -var-file="/tmp/tgtest/terragrunt-debug.tfvars.json" "/tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/mod"  prefix=[/tmp/tgtest]
DEBU[0001] Initializing remote state for the s3 backend  prefix=[/tmp/tgtest]
Remote state S3 bucket terragrunt-backend-s3-test000 does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y
DEBU[0006] Create S3 bucket with retry terragrunt-backend-s3-test000  prefix=[/tmp/tgtest]
DEBU[0006] Creating S3 bucket terragrunt-backend-s3-test000  prefix=[/tmp/tgtest]
DEBU[0007] Created S3 bucket terragrunt-backend-s3-test000  prefix=[/tmp/tgtest]
DEBU[0007] Waiting for bucket terragrunt-backend-s3-test000 to be created  prefix=[/tmp/tgtest]
DEBU[0007] S3 bucket terragrunt-backend-s3-test000 created.  prefix=[/tmp/tgtest]
DEBU[0007] Enabling root access to S3 bucket terragrunt-backend-s3-test000  prefix=[/tmp/tgtest]
ERRO[0013] Create S3 bucket with retry terragrunt-backend-s3-test000 returned an error: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors. Sleeping for 10s and will try again.  prefix=[/tmp/tgtest]
DEBU[0023] Create S3 bucket with retry terragrunt-backend-s3-test000  prefix=[/tmp/tgtest]
DEBU[0023] Creating S3 bucket terragrunt-backend-s3-test000  prefix=[/tmp/tgtest]
DEBU[0024] Looks like you're already creating bucket terragrunt-backend-s3-test000 at the same time. Will not attempt to create it again.  prefix=[/tmp/tgtest]
DEBU[0024] Waiting for bucket terragrunt-backend-s3-test000 to be created  prefix=[/tmp/tgtest]
DEBU[0024] S3 bucket terragrunt-backend-s3-test000 created.  prefix=[/tmp/tgtest]
WARN[0024] Versioning is not enabled for the remote state S3 bucket terragrunt-backend-s3-test000. We recommend enabling versioning so that you can roll back to previous versions of your Terraform state in case of error.  prefix=[/tmp/tgtest]
DEBU[0024] Running command: terraform init               prefix=[/tmp/tgtest]

❯ cat terragrunt-debug.tfvars.json
{}%

❯ aws s3api get-bucket-versioning --bucket terragrunt-backend-s3-test000

❯ aws s3api get-bucket-encryption --bucket terragrunt-backend-s3-test000

An error occurred (ServerSideEncryptionConfigurationNotFoundError) when calling the GetBucketEncryption operation: The server side encryption configuration was not found

❯ aws s3api get-bucket-policy --bucket terragrunt-backend-s3-test000

An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist

Workarounds

Using AWS Credential Environment variables

If you use standard AWS credential env vars (i.e. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) instead of using an AWS Profile (i.e. AWS_PROFILE and/or AWS_DEFAULT_PROFILE env vars) the bucket creation and parameters are set correctly, as expected, without error.
NOTE: It does NOT matter where the AWS cred env vars are sourced from (e.g. AWS IAM User, AWS IAM Role assumption, AWS SSO generated CLI Creds)

Terragrunt Apply (first-run with backend S3 bucket creation)

❯ terragrunt apply --terragrunt-debug --terragrunt-log-level debug
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Running command: terraform --version          prefix=[/tmp/tgtest]
DEBU[0001] Terraform version: 0.14.10
DEBU[0001] Reading Terragrunt config file at /tmp/tgtest/terragrunt.hcl
DEBU[0001] Did not find any locals block: skipping evaluation.
DEBU[0001] Downloading Terraform configurations from file:///tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY
DEBU[0001] Copying files from /tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module
DEBU[0001] Setting working directory to /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module
DEBU[0001] Generated file /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module/backend.tf.  prefix=[/tmp/tgtest]
INFO[0001] Debug mode requested: generating debug file terragrunt-debug.tfvars.json in working dir /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module  prefix=[/tmp/tgtest]
DEBU[0001] The following variables were detected in the terraform module:  prefix=[/tmp/tgtest]
DEBU[0001] []                                            prefix=[/tmp/tgtest]
DEBU[0001] Variables passed to terraform are located in "/tmp/tgtest/terragrunt-debug.tfvars.json"  prefix=[/tmp/tgtest]
DEBU[0001] Run this command to replicate how terraform was invoked:  prefix=[/tmp/tgtest]
DEBU[0001] 	terraform apply -var-file="/tmp/tgtest/terragrunt-debug.tfvars.json" "/tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module"  prefix=[/tmp/tgtest]
DEBU[0001] Initializing remote state for the s3 backend  prefix=[/tmp/tgtest]
Remote state S3 bucket terragrunt-backend-s3-test001 does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y
DEBU[0005] Create S3 bucket with retry terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0005] Creating S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0006] Created S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0006] Waiting for bucket terragrunt-backend-s3-test001 to be created  prefix=[/tmp/tgtest]
DEBU[0006] S3 bucket terragrunt-backend-s3-test001 created.  prefix=[/tmp/tgtest]
DEBU[0006] Enabling root access to S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0007] Enabled root access to bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0007] Enabling enforced TLS access for S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0007] Enabled enforced TLS access for bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0007] Blocking all public access to S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0008] Blocked all public access to S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0008] No tags specified for bucket terragrunt-backend-s3-test001.  prefix=[/tmp/tgtest]
DEBU[0008] Enabling versioning on S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0008] Enabled versioning on S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0008] Enabling bucket-wide SSE on AWS S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0008] Enabled bucket-wide SSE on AWS S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0008] Access Logging is disabled for the remote state AWS S3 bucket terragrunt-backend-s3-test001  prefix=[/tmp/tgtest]
DEBU[0008] Running command: terraform init               prefix=[/tmp/tgtest]

❯ cat terragrunt-debug.tfvars.json
{}%

❯ aws s3api get-bucket-versioning --bucket terragrunt-backend-s3-test001
{
    "Status": "Enabled"
}

❯ aws s3api get-bucket-encryption --bucket terragrunt-backend-s3-test001
{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms"
                },
                "BucketKeyEnabled": false
            }
        ]
    }
}

❯ aws s3api get-bucket-policy --bucket terragrunt-backend-s3-test001
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AllowTLSRequestsOnly\",\"Effect\":\"Deny\",\"Principal\":\"*\",\"Action\":\"s3:*\",\"Resource\":[\"arn:aws:s3:::terragrunt-backend-s3-test001\",\"arn:aws:s3:::terragrunt-backend-s3-test001/*\"],\"Condition\":{\"Bool\":{\"aws:SecureTransport\":\"false\"}}}]}"
}

Using Terragrunt iam_role config property to assume the role instead of an AWS Profile

NOTE: This method will not work with assumption of SSO IAM Roles as they are only assumable by the AWS SSO service!

Terragrunt.hcl

iam_role = "arn:aws:iam::{REDACTED}:role/okta-AWS-CloudAdmin"

terraform {
  source = ".//test_module"
}

remote_state {
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  backend = "s3"
  config = {
    encrypt = true
    bucket  = "terragrunt-backend-s3-test005"
    key     = "terraform.tfstate"
    region  = "us-west-2"

    s3_bucket_tags = {
        Owner       = "foo"
        Environment = "test"
    }
  }
}

test_module/test.tf

# Does nothing but generate an output based on caller identity
provider "aws" {
  region = "us-west-2"
}

data "aws_caller_identity" "test" {}

output "caller_id" {
  value = data.aws_caller_identity.test
}

Terragrunt Apply (first-run with backend S3 bucket creation)

❯ terragrunt apply --terragrunt-log-level debug
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Running command: terraform --version          prefix=[/tmp/tgtest]
DEBU[0001] Terraform version: 0.14.10
DEBU[0001] Reading Terragrunt config file at /tmp/tgtest/terragrunt.hcl
DEBU[0001] Did not find any locals block: skipping evaluation.
DEBU[0001] Assuming IAM role arn:aws:iam::{REDACTED}:role/okta-AWS-CloudAdmin
DEBU[0002] Downloading Terraform configurations from file:///tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY
DEBU[0002] Copying files from /tmp/tgtest into /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module
DEBU[0002] Setting working directory to /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module
DEBU[0002] Generated file /tmp/tgtest/.terragrunt-cache/I4xXOEwr4vXM7HkWXn2JJ0W772M/lJw6GWXuZS6lhlJTWNpiqodZCtY/test_module/backend.tf.  prefix=[/tmp/tgtest]
DEBU[0002] Initializing remote state for the s3 backend  prefix=[/tmp/tgtest]
Remote state S3 bucket terragrunt-backend-s3-test005 does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y
DEBU[0006] Create S3 bucket with retry terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0006] Creating S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0007] Created S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0007] Waiting for bucket terragrunt-backend-s3-test005 to be created  prefix=[/tmp/tgtest]
DEBU[0007] S3 bucket terragrunt-backend-s3-test005 created.  prefix=[/tmp/tgtest]
DEBU[0007] Enabling root access to S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0008] Enabled root access to bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0008] Enabling enforced TLS access for S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0008] Enabled enforced TLS access for bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0008] Blocking all public access to S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0008] Blocked all public access to S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
  1 iam_role = "arn:aws:iam::{REDACTED}:role/okta-AWS-CloudAdmin"
DEBU[0008] Tagging S3 bucket with map[Environment:test Owner:foo]  prefix=[/tmp/tgtest]
DEBU[0008] Tagged S3 bucket with map[Environment:test Owner:foo]  prefix=[/tmp/tgtest]
DEBU[0008] Enabling versioning on S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0009] Enabled versioning on S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0009] Enabling bucket-wide SSE on AWS S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0009] Enabled bucket-wide SSE on AWS S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0009] Access Logging is disabled for the remote state AWS S3 bucket terragrunt-backend-s3-test005  prefix=[/tmp/tgtest]
DEBU[0009] Running command: terraform init               prefix=[/tmp/tgtest]

❯ aws s3api get-bucket-tagging --bucket terragrunt-backend-s3-test005
{
    "TagSet": [
        {
            "Key": "Environment",
            "Value": "test"
        },
        {
            "Key": "Owner",
            "Value": "foo"
        }
    ]
}

❯ aws s3api get-bucket-versioning --bucket terragrunt-backend-s3-test005
{
    "Status": "Enabled"
}

❯ aws s3api get-bucket-policy --bucket terragrunt-backend-s3-test005
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AllowTLSRequestsOnly\",\"Effect\":\"Deny\",\"Principal\":\"*\",\"Action\":\"s3:*\",\"Resource\":[\"arn:aws:s3:::terragrunt-backend-s3-test005\",\"arn:aws:s3:::terragrunt-backend-s3-test005/*\"],\"Condition\":{\"Bool\":{\"aws:SecureTransport\":\"false\"}}}]}"
}

❯ aws s3api get-bucket-encryption --bucket terragrunt-backend-s3-test005
{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms"
                },
                "BucketKeyEnabled": false
            }
        ]
    }
}

Related: #1129, #1537
Possibly related: #1369

[jeff51419]

I had an issue as same as you. At first i believe is the permission issue on aws arn.
Terragrunt create resource correctly, but state bucket is public without versioning and server-side encryption.

Terragrunt.hcl

locals {
  # Automatically load account-level variables
  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  
  short_region    = local.region_vars.locals.short_region
  aws_region      = local.region_vars.locals.aws_region
  
  aws_profile     = local.account_vars.locals.aws_profile
  aws_assume_role = local.account_vars.locals.aws_assume_role
}
# Generate an AWS provider block
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<EOF
provider "aws" {
  shared_credentials_file = "~/.aws/credentials"

  # Only these AWS Account IDs may be operated on by this template
  profile                 = "${local.aws_profile}"
  region                  = "${local.aws_region}"
  assume_role {
    role_arn = "${local.aws_assume_role}"
  }
}
provider "random" {
}
EOF
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = "${local.account_name}-${local.short_region}-test-terraform-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = local.aws_region
    profile        = local.aws_profile
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

account.hcl

# Set account-wide variables. These are automatically pulled in to configure the remote state bucket in the root
# terragrunt.hcl configuration.
locals {
  account_name      = "account1"
  aws_account_name  = "aws-account1"
  aws_profile       = "aad-sso-account1"
  aws_assume_role   = "arn:aws:iam::123456789012:role/terrgrunt-role"
}

region.hcl

# Set common variables for the region. This is automatically pulled in in the root terragrunt.hcl configuration to
# configure the remote state bucket and pass forward to the child modules as inputs.
locals {
  aws_region    = "ap-southeast-1"
  short_region  = "aps1"
}

$ terragrunt plan --terragrunt-debug --terragrunt-log-level debug

.....
Remote state S3 bucket account1-aps1-test-terraform-state does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y
DEBU[0006] Create S3 bucket with retry account1-aps1-test-terraform-state  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0006] Creating S3 bucket account1-aps1-test-terraform-state  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0008] Created S3 bucket account1-aps1-test-terraform-state  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0008] Waiting for bucket account1-aps1-test-terraform-state to be created  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0008] S3 bucket account1-aps1-test-terraform-state created.  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0008] Enabling root access to S3 bucket account1-aps1-test-terraform-state  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
ERRO[0013] Create S3 bucket with retry account1-aps1-test-terraform-state returned an error: NoCredentialProviders: no valid providers in chain. Deprecated.
  For verbose messaging see aws.Config.CredentialsChainVerboseErrors. Sleeping for 10s and will try again.  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0023] Create S3 bucket with retry account1-aps1-test-terraform-state  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0023] Creating S3 bucket account1-aps1-test-terraform-state  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
DEBU[0024] Looks like you're already creating bucket account1-aps1-test-terraform-state at the same time. Will not attempt to create it again.  prefix=[/Users/jeff.yen/code/stp-devops-terragrunt/terragrunt/account1/test/ap-southeast-1/BE/vpc-db]
.....

Follow our company policy, we don't have own aws account. We use azure sso, then choose assume_role.
SO this issue cause I need to edit remote state S3 bucket after terragrunt apply.

[yorinasub17]

Fix released as https://github.com/gruntwork-io/terragrunt/releases/tag/v0.29.1. Binaries should show up shortly.

[crablab]

I am still seeing issues in 0.29.1 (and Tf 0.15.1) so I am not sure whether it's best to open a new issue (like #1661) or just comment here?

The brief synopsis is Terragrunt seems to be ignoring all credentials except the [default] block in ~/.aws/credentials. Hence, unless I manually set that block I get the "state bucket does not exist" etc.

[yorinasub17]

Can you share how you are invoking terragrunt? The fix in v0.29.1 is to properly honor either the AWS_PROFILE environment variable, or using the profile attribute on the remote_state config.

[crablab]

I was setting profile to [default] in the remote_state config but I kinda expected setting the environment variable to take precedence over that? This is probably my mistake and intended behaviour then.

[yorinasub17]

Ah because of the way the SDK works, the profile config in remote_state actually has precedence because we explicitly set it in the SDK config when you provide it, and the SDK only looks at the environment variable if that is not set.

[crablab]

🤦 My bad. Thanks for the explanation 😄