rds resources tag looks ups happen in wrong account when assuming cross account iam roles in aws

[sgarlick987]

Terraform Version

0.9.11

Affected Resource(s)

• aws_db_instance
• aws_db_parameter_group
• aws_db_subnet_group
• possibly other taggable rds resources

Debug Output

https://gist.github.com/sgarlick987/961415aa79fda5e804b724e7a6627230

I've included a few things here. One is the aws api call that fails, next is an example of a correct resource being looked up but then incorrectly being looked up when doing the tag comparison, and finally the error output of an apply.

Expected Behavior

When using cross account roles we expect the rds/ListTagsForResource api to use the account the database resources are in and where we have assumed the role.

Actual Behavior

When using cross account roles rds/ListTagsForResource gets called against the resources in the origin account instead of the target account we've assumed the role in. All other calls seem to get the correct account and arn so this doesn't affect it trying to recreate a database, but instead just shows that the existing ones have no tags and attempts to readd them (and fails on apply).

Steps to Reproduce

See our setup below, but the basic idea is to have 2 aws accounts with cross account roles setup and a rds resource created in the target account. Use the origin account role to assume the target account role, and run a plan. At this point it'll come back with having 0 tags, and adding back any tags you had (and that still currently exist). When trying to apply this plan it fails.

Important Factoids

This is our setup.
We run a jenkins master on kubernetes (and k8 on aws ec2 vpc), which runs terraform jobs in k8 pods. We use the kube2iam addon to proxy IAM roles to the pods terraform is executing in.
We have one account called "infra" where jenkins and k8 are run from on an ec2 and other accounts called "dev" and "prod". The k8 ec2 has an instance profile applied that has permission to assume other roles in "dev", "prod", and "infra". We configure kube2iam with these assumable roles and run a terraform job inside a pod its configured for. The dev and prod accounts show the issue because their tag lookups are happening in infra (assuming the role in infra doesn't show the problem since the assumed role is in the same account as the origin role).

We hold remote state in a s3 bucket of each target account and everything works fine, correct state gets pulled, correct dynamo lock is applied, all resources refresh fine and come back with no change, other than rds tags. I've confirmed the correct role comes back from the metadata proxy, the ec2 roles are able to be assumed fine.

I did see hashicorp/terraform#9821 which seems similar, but the user says their issue was cause by lack of permissions which doesn't seem to be the same in my case.

Let me know if there's any other information I might have missed that will help out. Thanks.

[sgarlick987]

Just to note, I've gotten around this for now by adding

lifecycle { ignore_changes = ["tags"] }

to those resources so it doesn't even bother doing the check and can apply everything else fine.

[potto007]

We've run into this at Ticketmaster where our CI is performed by runners on Kubernetes. We are doing AWS cross-account Terraform builds with assume-role applied via pod annotations. Jobs that do anything related to tagging with RDS resources fail.

Debug output logs from Terraform show that assume-role is happening, and the correct account is being hit for all resources in the RDS-related jobs, except for calls to add-tags-to-resource, list-tags-for-resource, and remove-tags-from-resource, which specifically reference an ARN. The ARN being used includes the AccountID of the AWS account from which the k8s pod is executing the Terraform job.

[bflad]

There have been a few technical debt pull requests merged over the last few releases of the AWS provider that switch RDS resources from using manually built ARNs based on provider initialization information to using the ARNs provided back by the RDS API. If this issue is still occurring after version 1.31.0 of the AWS provider, please reach out. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!