Terraform 0.6.11 fails to apply TTL changes to Route53 DNS entries for Weighted Round Robin

[eedwardsdisco]

I have 3 Route53 resource records which are part of a Weighted Set.

I am trying to change the TTL value on them, but it appears that the way that Terraform batches these changes, I'm unable to change the TTL.

This breaks the ability to use Terraform to manage DNS for these resources.

Examples

Before

resource "aws_route53_record" "cluster_endpoint" {
    zone_id = "${var.dns_domain_zone_id}"
    name = "consul-${var.cluster_name}"
    type = "A"
    ttl = "300"
    weight = "33"
    set_identifier = "${element(aws_instance.servers.*.tags.Name, count.index)}"
    count = "${var.server_count}"
    records = ["${element(aws_instance.servers.*.private_ip, count.index)}"]
}

After

resource "aws_route53_record" "cluster_endpoint" {
    zone_id = "${var.dns_domain_zone_id}"
    name = "consul-${var.cluster_name}"
    type = "A"
    ttl = "60"
    weight = "33"
    set_identifier = "${element(aws_instance.servers.*.tags.Name, count.index)}"
    count = "${var.server_count}"
    records = ["${element(aws_instance.servers.*.private_ip, count.index)}"]
}

Terraform Plan

Refreshing Terraform state prior to plan...

<snip>

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

~ module.consul_cluster.aws_route53_record.cluster_endpoint.0
    ttl: "300" => "60"

~ module.consul_cluster.aws_route53_record.cluster_endpoint.1
    ttl: "300" => "60"

~ module.consul_cluster.aws_route53_record.cluster_endpoint.2
    ttl: "300" => "60"


Plan: 0 to add, 3 to change, 0 to destroy.

Terraform Apply

<snip>
module.consul_cluster.aws_route53_record.cluster_endpoint.2: Modifying...
  ttl: "300" => "60"
module.consul_cluster.aws_route53_record.cluster_endpoint.0: Modifying...
  ttl: "300" => "60"
module.consul_cluster.aws_route53_record.cluster_endpoint.1: Modifying...
  ttl: "300" => "60"
Error applying plan:

3 error(s) occurred:

* aws_route53_record.cluster_endpoint.2: InvalidChangeBatch: RRSet with DNS name consul-dev.<snip>. and type A, SetIdentifier consul-dev-003 cannot be created as weighted sets must contain the same TTL.
    status code: 400, request id: b463e47e-db50-11e5-85ab-1d2ea4d3f776
* aws_route53_record.cluster_endpoint.0: InvalidChangeBatch: RRSet with DNS name consul-dev.<snip>. and type A, SetIdentifier consul-dev-001 cannot be created as weighted sets must contain the same TTL.
    status code: 400, request id: b473ea25-db50-11e5-a453-b3ca2aca1a44
* aws_route53_record.cluster_endpoint.1: InvalidChangeBatch: RRSet with DNS name consul-dev.<snip>. and type A, SetIdentifier consul-dev-002 cannot be created as weighted sets must contain the same TTL.
    status code: 400, request id: b50d434e-db50-11e5-b2f4-a7903934338a

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[eedwardsdisco]

@catsby any idea if this is addressed by the changes in #6954 ?

[catsby]

@eedwardsdisco Thank you for reporting this. Unfortunately #6954 does not address this problem.

The problem here is that AWS won't allow different TTLs for records in the same weighted set. What's happening here is Terraform is handling each of these resources separately, so it's issuing $count UPSERT commands to AWS, instead of doing it in a single bulk update. Unfortunately there isn't much I can do here; each resource in Terraform has no real knowledge of any other resource, and we have no way of bundling this kind of operation together at this time.

A workaround I've found is to change the set_identifier at the same time you're changing the TTL. While this works quickly, technically it's a ForceNew operation and so Terraform destroys your old records and quickly creates the new ones. I've found this to be fast, however, it could mean a blip in availability for you. Hopefully this is a rare occurrence and a blip is acceptable.

With regret, there isn't much I can do here. I'm going to close this issue, but feel free to follow up with more questions if you have any.

[robkinyon]

Unfortunately there isn't much I can do here; each resource in Terraform has no real knowledge of any other resource, and we have no way of bundling this kind of operation together at this time.

I'm going to (very gently!) call shenanigans on this one. While it is true that each resource in Terraform normally has no real knowledge of any other resource (other than the edges in the dependency tree), there's no reason for Terraform not to batch all DNS changes together as a single change batch. Not only is this faster, but it also is more audit-friendly because all the changes are marked as having been done together in CloudTrail.

This could done by creating a meta-resource that acts as a collector for all DNS changes within the working tree of resources. This meta-resource would have the superset of all edges and just be a collection of DNS changes. Then, when the tree is walked for what API calls to make, it would generate a single batch of DNS changes at the right time in the tree.

[so0k]

for now, changing the TTL on 1 of the weighted record set records in the Amazon Console automatically updates all of them.. so you can apply this through the UI (if no downtime should be tolerated and you have the option of using the UI)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.