Unexpected changes with aws_emr_cluster when using instance_group list

[jhart-r7]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

12:34:41 Terraform v0.11.7
12:34:42 + provider.aws v1.56.0
12:34:42 + provider.logentries v1.0.0
12:34:42 + provider.random v2.0.0
12:34:42 + provider.template v2.0.0

Affected Resource(s)

• aws_emr_cluster

Terraform Configuration Files

resource "aws_emr_cluster" "this_emr_cluster" {
    ...
    instance_group = "${var.instance_groups}"
    ...
}

And variables:

variable "instance_groups" {
    type = "list"
    default = [
        {
            name = "MasterInstanceGroup"
            instance_role = "MASTER"
            instance_type = "m3.xlarge"
            instance_count = 1
        },
        {
            name = "CoreInstanceGroup"
            instance_role = "CORE"
            instance_type = "r4.xlarge"
            instance_count = 2
            ebs_config = [
                {
                    iops = 0
                    size = 100
                    type = "gp2"
                    volumes_per_instance = 1
                }
            ]
        },
        {
            name = "TaskInstanceGroup"
            instance_role = "TASK"
            instance_type = "r4.xlarge"
            instance_count = 3
            ebs_config = [
                {
                    iops = 0
                    size = 100
                    type = "gp2"
                    volumes_per_instance = 1
                }
            ]
        },
    ]
}

Debug Output

Expected Behavior

No changes.

Actual Behavior

Changes:

12:39:29   ~ module.this_emr.aws_emr_cluster.this_emr_cluster
12:39:29       instance_group.#:                                                     "3" => "3"
12:39:29       instance_group.1929828997.autoscaling_policy:                         "" => ""
12:39:29       instance_group.1929828997.bid_price:                                  "" => ""
12:39:29       instance_group.1929828997.instance_count:                             "1" => "0"
12:39:29       instance_group.1929828997.instance_role:                              "MASTER" => ""
12:39:29       instance_group.1929828997.instance_type:                              "m3.xlarge" => ""
12:39:29       instance_group.1929828997.name:                                       "" => ""
12:39:29       instance_group.2262856773.autoscaling_policy:                         "" => ""
12:39:29       instance_group.2262856773.bid_price:                                  "" => ""
12:39:29       instance_group.2262856773.ebs_config.#:                               "" => "1"
12:39:29       instance_group.2262856773.ebs_config.1681599472.iops:                 "" => "0"
12:39:29       instance_group.2262856773.ebs_config.1681599472.size:                 "" => "100"
12:39:29       instance_group.2262856773.ebs_config.1681599472.type:                 "" => "gp2"
12:39:29       instance_group.2262856773.ebs_config.1681599472.volumes_per_instance: "" => "1"
12:39:29       instance_group.2262856773.id:                                         "" => <computed>
12:39:29       instance_group.2262856773.instance_count:                             "" => "2"
12:39:29       instance_group.2262856773.instance_role:                              "" => "CORE"
12:39:29       instance_group.2262856773.instance_type:                              "" => "r4.xlarge"
12:39:29       instance_group.2262856773.name:                                       "" => "CoreInstanceGroup"
12:39:29       instance_group.2933868783.autoscaling_policy:                         "" => ""
12:39:29       instance_group.2933868783.bid_price:                                  "" => ""
12:39:29       instance_group.2933868783.ebs_config.#:                               "" => "1"
12:39:29       instance_group.2933868783.ebs_config.1681599472.iops:                 "" => "0"
12:39:29       instance_group.2933868783.ebs_config.1681599472.size:                 "" => "100"
12:39:29       instance_group.2933868783.ebs_config.1681599472.type:                 "" => "gp2"
12:39:29       instance_group.2933868783.ebs_config.1681599472.volumes_per_instance: "" => "1"
12:39:29       instance_group.2933868783.id:                                         "" => <computed>
12:39:29       instance_group.2933868783.instance_count:                             "" => "3"
12:39:29       instance_group.2933868783.instance_role:                              "" => "TASK"
12:39:29       instance_group.2933868783.instance_type:                              "" => "r4.xlarge"
12:39:29       instance_group.2933868783.name:                                       "" => "TaskInstanceGroup"
12:39:29       instance_group.303465708.autoscaling_policy:                          "" => ""
12:39:29       instance_group.303465708.bid_price:                                   "" => ""
12:39:29       instance_group.303465708.instance_count:                              "3" => "0"
12:39:29       instance_group.303465708.instance_role:                               "TASK" => ""
12:39:29       instance_group.303465708.instance_type:                               "r4.xlarge" => ""
12:39:29       instance_group.303465708.name:                                        "" => ""
12:39:29       instance_group.352319499.autoscaling_policy:                          "" => ""
12:39:29       instance_group.352319499.bid_price:                                   "" => ""
12:39:29       instance_group.352319499.instance_count:                              "2" => "0"
12:39:29       instance_group.352319499.instance_role:                               "CORE" => ""
12:39:29       instance_group.352319499.instance_type:                               "r4.xlarge" => ""
12:39:29       instance_group.352319499.name:                                        "" => ""
12:39:29       instance_group.3945898279.autoscaling_policy:                         "" => ""
12:39:29       instance_group.3945898279.bid_price:                                  "" => ""
12:39:29       instance_group.3945898279.ebs_config.#:                               "" => <computed>
12:39:29       instance_group.3945898279.id:                                         "" => <computed>
12:39:29       instance_group.3945898279.instance_count:                             "" => "1"
12:39:29       instance_group.3945898279.instance_role:                              "" => "MASTER"
12:39:29       instance_group.3945898279.instance_type:                              "" => "m3.xlarge"
12:39:29       instance_group.3945898279.name:                                       "" => "MasterInstanceGroup"

Steps to Reproduce

1. terraform plan or apply with an aws_emr_cluster that uses instance_group

Important Factoids

Even after an apply, subsequent plans show more of the same changes.

References

#6498 may be relevant, as this change was not happening with 1.55.0 and started only after 1.56.0 (and template 2.0.0). I believe instance_group is getting reordered as part of the plan by list/set conversions.

[jhart-r7]

I have also tried specifying 3 separate instance_group:

instance_group {
        name = "CoreInstanceGroup"
        instance_role = "CORE"
        instance_type = "r4.xlarge"
        instance_count = 2
        ebs_config = [
            {
               iops = 0
               size = 100
               type = "gp2"
               volumes_per_instance = 1
            }
        ]
    }

    instance_group {
        name = "TaskInstanceGroup"
        instance_role = "TASK"
        instance_type = "r4.xlarge"
        instance_count = 3
        ebs_config = [
           {
              iops = 0
              size = 100
              type = "gp2"
              volumes_per_instance = 1
           }
        ]
     }

    instance_group {
        name = "MasterInstanceGroup"
        instance_role = "MASTER"
        instance_type = "m3.xlarge"
        instance_count = 1
    }

I even tried re-ordering the above without luck -- the plan still shows changes; Perhaps I'd need to apply and then 🤞 that the subsequent plan or check-plan-empty succeeded.

[bflad]

The fix for this (#7324) has been merged and will release with version 1.57.0 of the Terraform AWS provider, likely later today.

[jhart-r7]

👍 thank you!.

As a note I was able to work around this problem, it seems, by removing the name attribute from our EMR configuration.

[bflad]

This has been released in version 1.57.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!