Tags are removed from EBS volumes

[mdelagrange]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.14.4

• provider registry.terraform.io/hashicorp/aws v3.23.0

Affected Resource(s)

• aws_ebs_volume
• aws_volume_attachment
• aws_instance

Terraform Configuration Files

resource "aws_instance" "web" {
  ami               = "ami-089668cd321f3cf82"
  instance_type     = "t3.micro"
  availability_zone = "us-west-2a"

  lifecycle {
    ignore_changes = [
      root_block_device,
    ]
  }
}

resource "aws_ebs_volume" "example1" {
  availability_zone = "us-west-2a"

  tags = {
    Name = "example1"
  }
}

resource "aws_volume_attachment" "example1" {
  device_name = "/dev/sdh"
  volume_id   = aws_ebs_volume.example1.id
  instance_id = aws_instance.web.id
}

Expected Behavior

The first apply and subsequent applies result in an EBS volume with a "Name" tag set to "example1".

Actual Behavior

The first apply creates the volume with the correct tag, but subsequent applies alternatively remove and recreate the tag.

Steps to Reproduce

1. terraform apply. The volume will be created correctly with the "Name" tag set to "example1".
2. terraform apply again. The "Name" tag will be removed from the volume.
3. terraform apply again. The "Name" tag will be added to the volume.
4. terraform apply again. The "Name" tag will be removed from the volume.
5. terraform apply again. The "Name" tag will be added to the volume.

Important Factoids

Applies that remove the tag from the volume have the following plan. This is unexpected, because the instance resource does not specify volume_tags:

Terraform will perform the following actions:

  # aws_instance.web will be updated in-place
  ~ resource "aws_instance" "web" {
        id                           = "i-06ccbb7893d13d8cf"
        tags                         = {}
      ~ volume_tags                  = {
          - "Name" = "example1" -> null
        }
        # (26 unchanged attributes hidden)

        # (5 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Removing root_block_device from the ignored changes on aws_instance will prevent volume tags from being removed. However, I'm seeing other examples in my Terraform environment of tags being removed from volumes which do not ignore changes to root_block_device. This was just the first example that I could reproduce simply.

[ewbankkit]

Similar:

• Overwrite tags to an EBS attached to an EC2 instance when volume_tags is specified #5878
• Terraform volume tags changes on alternate run if defined in both aws_instance and aws_ebs_volume resources  #5609
• Alternating tags in volume #5080
• aws_instance volume_tags conflict with aws_ebs_volume tags for attached volumes in a module #770
• aws_instance.volume_tags gets confused when > 1 volume attached #729
• volume_tags on instance (sometimes) clobbers attached EBS volume tags #884
• r/aws_instance: volume_tags works incorrectly with aws_volume_attachment resources #12225

[YakDriver]

We have merged a fix to the volume_tags issue in #15474. We have added tests to cover the issues observed. Please note that using volume_tags in aws_instance is not compatible with using tags in aws_ebs_volume. You need to use one or the other. Prior to this fix, even following this rule, you would encounter errors. Along with the fix, we've added tags to the root_block_device and ebs_block_device configuration blocks in aws_instance.

Now that the fix is in place, if you find any problems with volume_tags, let us know by opening a new issue.

[ghost]

This has been released in version 3.24.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[SpComb]

We're seeing something similar after upgrading terraform-provider-aws 3.19 -> 3.26: we have never used aws_instance volume_tags, but terraform apply shows all aws_instance resources as changed, with the volume_tags attribute being removed. We are also using ignore_changes = [root_block_device].

Applying the terraform change to the aws_instance resources will cause all tags to be removed from the EBS volumes attached to the EC2 instances. A second apply will notice that the tags are missing from the attached aws_ebs_volume resource, and restore them. Further applies will not show any changes - EDIT: which is different from the v3.23.0 issue here.

Is this a new issue caused by the #15474 fix in 3.24?

EDIT: See also #15474 (comment)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!