DNS record not updated when public IP changes due to instance type change

[ghost]

This issue was originally opened by @wayneworkman as hashicorp/terraform#19578. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.11.10

• provider.aws v1.46.0
• provider.template v1.0.0

I have an ec2 instance that gets assigned a public IP, and a route53 resource that creates a CNAME record pointing at that ec2 instance's dns name.

I updated the ec2 instance from a t3.nano to t3.micro. This resulted in the instance stopping, changing it's type, and then starting up. The instance's public IP changed due to the stop/start, but the route53 record was not updated with the new DNS name.

Expected Behavior

Terraform should have updated the dns record.

Actual Behavior

Terraform did not update the dns record.

How to reproduce

Run this, which makes an instance with a public IP and a DNS record for it.

resource "aws_instance" "bastion" {
  ami           = "${var.amis["debian9"]}"
  instance_type = "t2.micro"
  subnet_id = "${aws_subnet.public-subnet.id}"
  vpc_security_group_ids = ["${aws_security_group.sg-ssh.id}"]
  associate_public_ip_address = true
  key_name = "${aws_key_pair.ssh-key.key_name}"
}

resource "aws_route53_record" "bastion-dns-record" {
  zone_id = "${var.zone_id}"
  name    = "fogbastion.${var.zone_name}"
  type    = "CNAME"
  ttl     = "300"
  records = ["${aws_instance.bastion.public_dns}"]
}

Then change the instance type to something else, and apply.

[jmery]

+1 on this issue. Hit the problem just a few mins ago. This issue was the first hit in search. It's more than just CNAME records though. I experienced it with an A record type.

[kshpytsya]

Use of aws_eip seems to work well as a workaround for this issue.

[ZMI-JesseMcAllister]

This issue persists in Terraform v0.12.1 with v2.13.0 of the AWS provider.

[karthikbeam]

Experienced this as well. This is likely because the instance reboots (or stop starts) to effect the instance size change and that changes the public ip (and thus the dns name).

Terraform v0.12.6
terraform-provider-aws_v2.23.0_x4

[karthikbeam]

The "workaround" is to terraform apply (resize) -> wait for it.. -> terraform apply (change dns record).

[manugarri]

Still an issue on v0.12.8 :(

[next-jesusmanuelnavarro]

Still an issue on terraform v0.12.24 with aws provider v2.55.0.

Indeed, when a reboot is involved (in my case was also because of an update on the instance type) terraform plan shows it doesn't expect ip-related properties of the instance to change (ip addresses themselves, associated DNS names, either public or private...) so resources that depend on them (also an associated CNAME in my case) will work on the wrong assumptions.

[next-jesusmanuelnavarro]

The "workaround" is to terraform apply (resize) -> wait for it.. -> terraform apply (change dns record).

I think I have a better one...

WORKAROUND: Use a "middle-man" data source.

So, instead of, say...

resource "aws_instance" "example" {
  (...)
}

resource "aws_route53_record" "example" {
 (...)
  records = [aws_instance.example.public_dns]
}

...use this:

resource "aws_instance" "example" {
  (...)
}

data "aws_instance" "example" {
  instance_id = aws_instance.example.id
}

resource "aws_route53_record" "example" {
 (...)
  records = [data.aws_instance.example.public_dns]
}

It seems the underlying "data" management code doesn't try to be as "clever" as others and forces a full "recalculation" of its contents only after the upstream resource (the aws instance, in this case) is fully updated.

This way, since the aws_instance.example is changed, the associated datasource gets "recalculated" and the final dependent resource (a DNS record in this case) ends up with the proper value instead of the old one.

[pipeworks-asmith]

Ran into this today with terraform 0.12.25 and AWS provider 2.70.0. Any hope of a real fix here?

[next-jesusmanuelnavarro]

Ran into this today with terraform 0.12.25 and AWS provider 2.70.0. Any hope of a real fix here?

Did my approach helped you?

In the meantime, I found the same kind of problem in other situations so, in the end, I went with a different workaround (over same logical basis). So... the problem is certain resources are not updated enough in advance? When are they going to be updated? By the end of terraform apply, right? (that's aligned with #6781 (comment)).

Instead of running Terraform twice, I move those resources to a different "stage" meant to be run secuentially. I.e.:

terraform/
  |-10-run-first/
  |    |-main.tf  # here I manage infra
  |-20-run-after/
       |-main.tf  # here I manage DNS records, knowing infra data will already be properly updated

(note 10-run-first and 20-run-after are not terraform modules but independent runs: you are meant to cd into 10-run-first and run terraform apply within, and then move to 20-run-after and run terraform apply there too).

Since I'm already using shared remote states, it's a matter of adding the proper outputs for the resources I'll be using later to 10-run-first and capturing them on 20-run-after by means of a terraform_remote_state data resource.

More cumbersome, but guaranteed to work no matter what.

[asaghri]

Hi, we still have this issue. Is it going to be a real fix for that ?
Tx !

[mariuskimmina]

+1

We are also still facing this issue.

[speller]

Four years have passed but the Terraform still produces invalid non-functional environments... Shame...
In my case, I'm trying to link a domain and an instance which user-data is updated on every deployment. The instance is restarted on every deployment and the IP address is changed, so the DNS record NEVER has a valid IP address! During the plan, it plans to update the IP address to the value existing now, during the planning, not the one that will be after rebooting, so when it reboots the instance, it obviously sets the old invalid IP address! Damn...

[speller]

The suggested workaround doesn't work in my case.

The instance is updated in-place in the plan:

module.mailhog.aws_instance.instance will be updated in-place
  ~ resource "aws_instance" "instance" {
        id                                   = "i-0e618912e507185b1"
      ~ user_data=(sensitive)
        # (35 unchanged attributes hidden)
        # (7 unchanged blocks hidden)
    }

But this doesn't trigger the route record update even if the instance is referred through the data object.

Sucks... I love Terraform and hate it. Hate for many issues of this kind which are not solved for many years... At the moment when you think that you've won the fight with a known issue, TF immediately brings you new problems...

[speller]

I think that the Terraform team should at least add a disclaimer somewhere that under some simple circumstances, TF with all its power is unable to link an AWS instance and route53 record. The documented bug is a feature (c).

[speller]

It seems the only workaround is to use the user_data_replace_on_change property.

[jklusis]

Hey! I'm also facing this issue in a similar case when changing the instance state via resource.

Attempted to use the data solution, but, unfortunately, it did not work out by itself in my case, probably due to me not changing the aws_instance itself. Found out a hack that, if the data attribute depends on a resource that will be directly changed (in my case – instance state), it will wait for it to be provisioned before reading the values.

Terraform:

resource "aws_instance" "this" {
  ami           = ...
  instance_type = ...

  associate_public_ip_address = true

...

  lifecycle {
    ignore_changes = [associate_public_ip_address]
  }
}

data "aws_instance" "this" {
  instance_id = aws_instance.this.id

  depends_on = [aws_ec2_instance_state.instance] # This is the addition that forces the data to force-refresh
}

resource "aws_ec2_instance_state" "instance" {
  instance_id = aws_instance.this.id
  state       = var.ec2_target_state
}

resource "aws_route53_record" "base" {
  count = var.ec2_target_state == "running" ? 1 : 0

  zone_id = var.dns_zone_id
  name    = <redacted>
  type    = "A"
  ttl     = 300
  records = [data.aws_instance.this.public_ip]
}

Plan:

  # data.aws_instance.this will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_instance" "this" {
...
      + instance_id                 = <redacted>
...
      + private_dns                 = (known after apply)
      + private_dns_name_options    = (known after apply)
      + private_ip                  = (known after apply)
      + public_dns                  = (known after apply)
      + public_ip                   = (known after apply)
...
    }

...

 # aws_ec2_instance_state.instance will be updated in-place
  ~ resource "aws_ec2_instance_state" "instance" {
        id          = "i-1234567890"
      ~ state       = "stopped" -> "running"
        # (2 unchanged attributes hidden)
    }

  # aws_route53_record.base[0] will be created
  + resource "aws_route53_record" "base" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "<redacted>"
      + records         = (known after apply) # This is what we like to see
      + ttl             = 300
      + type            = "A"
      + zone_id         = "<redacted>"
    }

Initially also tried depending directly within aws_route53_record but unfortunately it did not work. And, adding the dependency within aws_instance creates a circular dependency.

Agreed that ideally this would be resolved within Terraform itself.

[chrispenny]

Thank you for the workaround @jklusis! I was experiencing the same issue, and your solution worked for me 😄

[speller]

almost 6 years and nobody cares...

[ewbankkit]

🤞 this should be a case of setting public_dns and public_ip as newly-Computed if the instance stops and restarts during update (e.g. if the instance type changes).
See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#concepts-public-addresses.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.83.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.