provider/aws: TestAccAWSOpsworksCustomLayer tests failing inconsistently

[apparentlymart]

Over in #4272 @stack72 noticed a test failure on the aws_opsworks_custom_layer tests which can't be reproduced by the original submitter or by me, and seems at first glance to be orthogonal to that change.

It would appear that something is different about @stack72's configuration compared to @mrjefftang's and mine, but it's not clear what that might be.

In the past we've found issues where tests behave differently depending on what parts of the Opsworks UI have been used in a given AWS account, since the console seems to silently create or update objects in ways that the raw Opsworks APIs do not.

Test result:

TF_LOG=1 make testacc TEST=./builtin/providers/aws TESTARGS='-run=TestAccAWSOpsworksCustomLayer' 2>~/tf.log
==> Checking that code complies with gofmt requirements...
/Users/pstack/code/go/bin/stringer
go generate $(go list ./... | grep -v /vendor/)
TF_ACC=1 go test ./builtin/providers/aws -v -run=TestAccAWSOpsworksCustomLayer -timeout 120m
=== RUN   TestAccAWSOpsworksCustomLayer
--- FAIL: TestAccAWSOpsworksCustomLayer (15.54s)
    testing.go:154: Step 0 error: After applying this step, the plan was not empty:

        DIFF:

        UPDATE: aws_opsworks_stack.tf-acc
          default_subnet_id: "subnet-c0f30fb6" => ""
          vpc_id:            "vpc-b645c2d2" => ""

        STATE:

        aws_iam_instance_profile.opsworks_instance:
          ID = tf-4920485386043202654_opsworks_instance
          arn = arn:aws:iam::881237884953:instance-profile/tf-4920485386043202654_opsworks_instance
          name = tf-4920485386043202654_opsworks_instance
          path = /
          roles.# = 1
          roles.379301246 = tf-4920485386043202654_opsworks_instance

          Dependencies:
            aws_iam_role.opsworks_instance
        aws_iam_role.opsworks_instance:
          ID = tf-4920485386043202654_opsworks_instance
          arn = arn:aws:iam::881237884953:role/tf-4920485386043202654_opsworks_instance
          assume_role_policy = {
          "Version": "2008-10-17",
          "Statement": [
            {
              "Sid": "",
              "Effect": "Allow",
              "Principal": {
                "Service": "ec2.amazonaws.com"
              },
              "Action": "sts:AssumeRole"
            }
          ]
        }

          name = tf-4920485386043202654_opsworks_instance
          path = /
          unique_id = AROAIAJ32FXPLQX5ADRXO
        aws_iam_role.opsworks_service:
          ID = tf-4920485386043202654_opsworks_service
          arn = arn:aws:iam::881237884953:role/tf-4920485386043202654_opsworks_service
          assume_role_policy = {
          "Version": "2008-10-17",
          "Statement": [
            {
              "Sid": "",
              "Effect": "Allow",
              "Principal": {
                "Service": "opsworks.amazonaws.com"
              },
              "Action": "sts:AssumeRole"
            }
          ]
        }

          name = tf-4920485386043202654_opsworks_service
          path = /
          unique_id = AROAJIPRMGENLQCJBK2X2
        aws_iam_role_policy.opsworks_service:
          ID = tf-4920485386043202654_opsworks_service:tf-4920485386043202654_opsworks_service
          name = tf-4920485386043202654_opsworks_service
          policy = {
          "Statement": [
            {
              "Action": [
                "ec2:*",
                "iam:PassRole",
                "cloudwatch:GetMetricStatistics",
                "elasticloadbalancing:*",
                "rds:*"
              ],
              "Effect": "Allow",
              "Resource": ["*"]
            }
          ]
        }

          role = tf-4920485386043202654_opsworks_service

          Dependencies:
            aws_iam_role.opsworks_service
        aws_opsworks_custom_layer.tf-acc:
          ID = 98da61c7-1292-4838-ac73-a54d5c5452d0
          auto_assign_elastic_ips = false
          auto_assign_public_ips = true
          auto_healing = true
          custom_configure_recipes.# = 0
          custom_deploy_recipes.# = 0
          custom_instance_profile_arn =
          custom_json =
          custom_security_group_ids.# = 2
          custom_security_group_ids.4172965597 = sg-f0d1f088
          custom_security_group_ids.671131585 = sg-ffd1f087
          custom_setup_recipes.# = 0
          custom_shutdown_recipes.# = 0
          custom_undeploy_recipes.# = 0
          drain_elb_on_shutdown = true
          ebs_volume.# = 1
          ebs_volume.3575749636.iops = 0
          ebs_volume.3575749636.mount_point = /home
          ebs_volume.3575749636.number_of_disks = 2
          ebs_volume.3575749636.raid_level = 0
          ebs_volume.3575749636.size = 100
          ebs_volume.3575749636.type = gp2
          elastic_load_balancer =
          install_updates_on_boot = true
          instance_shutdown_timeout = 300
          name = tf-4920485386043202654
          short_name = tf-ops-acc-custom-layer
          stack_id = 0495ac71-3e8e-400d-a552-3bb0dbd5714a
          system_packages.# = 2
          system_packages.1368285564 = git
          system_packages.2937857443 = golang
          use_ebs_optimized_instances = false

          Dependencies:
            aws_opsworks_stack.tf-acc
            aws_security_group.tf-ops-acc-layer1
            aws_security_group.tf-ops-acc-layer2
        aws_opsworks_stack.tf-acc:
          ID = 0495ac71-3e8e-400d-a552-3bb0dbd5714a
          berkshelf_version = 3.2.0
          color =
          configuration_manager_name = Chef
          configuration_manager_version = 11.10
          custom_cookbooks_source.# = 1
          custom_json = {"key": "value"}
          default_availability_zone = us-east-1c
          default_instance_profile_arn = arn:aws:iam::881237884953:instance-profile/tf-4920485386043202654_opsworks_instance
          default_os = Amazon Linux 2014.09
          default_root_device_type = ebs
          default_ssh_key_name =
          default_subnet_id = subnet-c0f30fb6
          hostname_theme = Layer_Dependent
          manage_berkshelf = false
          name = tf-4920485386043202654
          region = us-east-1
          service_role_arn = arn:aws:iam::881237884953:role/tf-4920485386043202654_opsworks_service
          use_custom_cookbooks = false
          use_opsworks_security_groups = false
          vpc_id = vpc-b645c2d2

          Dependencies:
            aws_iam_instance_profile.opsworks_instance
            aws_iam_role.opsworks_service
        aws_security_group.tf-ops-acc-layer1:
          ID = sg-f0d1f088
          description = Managed by Terraform
          egress.# = 0
          ingress.# = 1
          ingress.490428346.cidr_blocks.# = 1
          ingress.490428346.cidr_blocks.0 = 0.0.0.0/0
          ingress.490428346.from_port = 8
          ingress.490428346.protocol = icmp
          ingress.490428346.security_groups.# = 0
          ingress.490428346.self = false
          ingress.490428346.to_port = -1
          name = tf-4920485386043202654-layer1
          owner_id = 881237884953
          tags.# = 0
          vpc_id = vpc-b645c2d2
        aws_security_group.tf-ops-acc-layer2:
          ID = sg-ffd1f087
          description = Managed by Terraform
          egress.# = 0
          ingress.# = 1
          ingress.490428346.cidr_blocks.# = 1
          ingress.490428346.cidr_blocks.0 = 0.0.0.0/0
          ingress.490428346.from_port = 8
          ingress.490428346.protocol = icmp
          ingress.490428346.security_groups.# = 0
          ingress.490428346.self = false
          ingress.490428346.to_port = -1
          name = tf-4920485386043202654-layer2
          owner_id = 881237884953
          tags.# = 0
          vpc_id = vpc-b645c2d2
FAIL
exit status 1
FAIL    github.com/hashicorp/terraform/builtin/providers/aws    15.554s

[catsby]

Hey @apparentlymart – odd, in our past few days of nightly ACC tests this has not failed. Are you still hitting it?

[catsby]

ohey – I think this is a test that needs or expects EC2 Classic, which explains why the Terraform nightly tests work, and my personal account with Classic passes, but my HashiCorp account without Classic fails.

If it's OK with you @apparentlymart we should just document that and close this

[mrjefftang]

I think a better alternative is to update the test to use a VPC instead of relying on the existence of EC2 Classic.

[apparentlymart]

@catsby so far only @stack72 was actually able to repro this error... I never had it fail.

If I'm understanding your suggestion correctly, you're suggesting that we just document that the tests are only expected to work on accounts with EC2-Classic enabled? In principle that seems fine to me, as long as getting new accounts with EC2-Classic is possible.

On the other hand, since EC2-Classic is deprecated maybe we should instead make all of the tests target VPC and document that Terraform is not guaranteed to support Opsworks completely on EC2-Classic? Not saying that we should intentionally break it, but that we'd not be acceptance testing it and so we can't guarantee that there won't be regressions in future.

[stack72]

This would make sense - my account doesn't have EC2-classic and it fails me for every single time

[catsby]

Hey Friends –

Thanks for all the thoughts and consideration here, but I've decided to close this. My reasoning being that Terraform is still committed to supporting EC2 Classic, and some of the OpsWorks tests are specifically designed to test that OpsWorks works in both Classic and VPCs, so this is a good idea to leave as is.

I'm committed to making all the AWS tests pass nightly, though it's a battle I continue to wage. You can see here that OpsWorks tests are passing, and I have a spreadsheet to show you that it's been passing for about 2 weeks or so, if you're interested 😄

you're suggesting that we just document that the tests are only expected to work on accounts with EC2-Classic enabled?

Yes, I'm suggesting that. We have a handful of other tests in Terraform that require Classic as well.

In principle that seems fine to me, as long as getting new accounts with EC2-Classic is possible.

Possible, I think. But difficult, I imagine. I personally have an account with Classic, I believe phinze does as well, and the account that runs our nightly acceptance tests does, so the responsibility will fall on us to make sure these pass.

On the other hand, since EC2-Classic is deprecated maybe we should instead make all of the tests target VPC and document that Terraform is not guaranteed to support Opsworks completely on EC2-Classic?

I don't think that's in the best interests of our users; Terraform supports EC2 Classic, so I'm going to keep tests around to ensure it's as solid as we can make it. I realize this may frustrate some of our core contributors, who can not run these Classic tests, and I sincerely apologize for that. We greatly appreciate your continued effort and contributions to Terraform! I'll take responsibility for the Classic tests, so if you come across one please ping me and I'll jump in 😄

Let me know if you have any other questions or concerns here.
Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.