Idempotency issue when using Provider tags

[gjkamstra]

Description

I am using provider tags as described in http://registry.terraform.io/providers/hashicorp/aws/latest/docs#default_tags-configuration-block. These tags don't get applied to the security groups, since they get created by EKS and not Terraform. Therefore I replicate these tags on the cluster_tags variable of the module. I do that by retrieving them from the provider settings using the aws_default_tags datasource.
However when i do that, terraform apply keeps telling me that it needs to update the tags. This is due to the cluster_tags getting merged with tags and then set on the eks resource.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]:
  18.28.0

• Terraform version:
  Terraform v1.2.7
  on darwin_amd64

• Provider version(s):

• provider registry.terraform.io/hashicorp/aws v4.27.0
• provider registry.terraform.io/hashicorp/cloudinit v2.2.0
• provider registry.terraform.io/hashicorp/kubernetes v2.12.0
• provider registry.terraform.io/hashicorp/null v3.1.1
• provider registry.terraform.io/hashicorp/tls v3.4.0

Reproduction Code [Required]

Example: master...gjkamstra:terraform-aws-eks:provider-tag-example

Steps to reproduce the behavior:
Execute terraform apply and say yes.
Execute it again, Terraform will say that there are still updates required to the tags. It will keep doing that

Expected behavior

No changes required, after we just did terraform apply

Actual behavior

Tag changes need to be applied

Terminal Output Screenshot(s)

plan
module.eks.aws_eks_cluster.this[0] will be updated in-place
  ~ resource "aws_eks_cluster" "this" {
        id                        = "gjktest-cluster"
        name                      = "gjktest-cluster"
      ~ tags                      = {
          + "Environment"                            = "MyFancyEnvironment"
            # (1 unchanged element hidden)
        }
        # (11 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Expanded plan:

{
    "change": {
        "actions": [
            "update"
        ],
        "before": {
            .......
            "tags": {
                "karpenter.sh/discovery/gjktest-cluster": "gjktest-cluster"
            },
            "tags_all": {
                "Environment": "MyFancyEnvironment",
                "karpenter.sh/discovery/gjktest-cluster": "gjktest-cluster"
            },
            ......
        },
        "after": {
            ...
            "tags": {
                "Environment": "MyFancyEnvironment",
                "karpenter.sh/discovery/gjktest-cluster": "gjktest-cluster"
            },
            "tags_all": {
                "Environment": "MyFancyEnvironment",
                "karpenter.sh/discovery/gjktest-cluster": "gjktest-cluster"
            },
            .....
}

Additional context

Issue comes from specifying the same tag on the provider as on the tags property (done since module merges cluster tags with tags and sets that to tags on the EKS resource. Documentation of the provider shows that you should not do this.
I have created two possible fixes.

Keep behavior as is, but remove provider tags from tags property.

You will still need to duplicate the provider tags on the cluster_tags variable to set the tag on the security group
master...gjkamstra:terraform-aws-eks:strip-provider-tags-from-tags

Automatically add the provider tags to the security groups

This way you no longer have to duplicate the provider tags on the cluster_tag variable. The module will retrieve them and set them by itself. I prefer this approach
master...gjkamstra:terraform-aws-eks:add-provider-tags-to-security-group

Please let me know which solution you prefer. I will then raise the pull request. Or if you have a different solution, that is fine too.

[bryantbiggs]

this is a well known issue with the use of default_tags hashicorp/terraform-provider-aws#18311

[gjkamstra]

Yes. Unfortunately I don't expect a solution in the provider anytime soon.
So way to solve/workaround it is to make sure the provider tags are not replicated on the tags property (see also that issue you mentioned). However the module merges the two tag lists and puts in on the tags property. I can therefore not control that myself.

I need to duplicate the provider tags on cluster_tags to get the tag on the security group that EKS creates. However, since the tag lists are merged, it then gets added on the EKS resource as well, which causes the issue.

There are two solutions above that will fix this, with different impact on users of this module.

[bryantbiggs]

thank you @gjkamstra for the detailed write up. I have been able to take some time to test and validate the details you have provided and I think we can most definitely support the 2nd option you have listed. This should be available starting in v18.30.0 once released in #2249

[antonbabenko]

This issue has been resolved in version 18.30.0 🎉

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.