[Bug]: waiting for API Gateway Domain Name (mydomain-name/6wnnm3i4kd) update: couldn't find resource (21 retries)

[bkeech]

Terraform Core Version

OpenTofu 1.8.3

AWS Provider Version

5.82.2

Affected Resource(s)

• aws_api_gateway_domain_name

Expected Behavior

Custom Domain Name policy should be updated

Actual Behavior

After the first initial apply is successful subsequent plans show that the Custom Domain resource policy has to be updated. When trying subsequent applies, the apply fails with:
Error: waiting for API Gateway Domain Name (mydomain-name/6wnnm3i4kd) update: couldn't find resource (21 retries)

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

data "aws_region" "region" {}
locals {
  region = data.aws_region.region.name
}

data "aws_vpc_endpoint" "apigw-vpce" {
  vpc_id       = var.vpc_id
  service_name = "com.amazonaws.${local.region}.execute-api"
  id           = var.vpc_endpoint_id
}

data "aws_route53_zone" "public_zone" {
  name         = lower(local.host_portion)
  private_zone = false
}

data "aws_route53_zone" "private_zone" {
  name         = lower(local.host_portion)
  private_zone = true
}

# Certificate and certificate validation records
resource "aws_acm_certificate" "cert" {
  domain_name       = local.fqdn
  validation_method = "DNS"
  tags              = var.tags
}


resource "aws_route53_record" "val-record" {
  for_each = {
    for dvo in aws_acm_certificate.cert.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = data.aws_route53_zone.public_zone.zone_id
}

resource "aws_acm_certificate_validation" "cert-val" {
  certificate_arn         = aws_acm_certificate.cert.arn
  validation_record_fqdns = [for record in aws_route53_record.val-record : record.fqdn]
}

resource "aws_route53_record" "lb-a-record" {
  name    = lower(local.fqdn)
  type    = "A"
  zone_id = data.aws_route53_zone.private_zone.zone_id

  # Used for multi-region deployments.
  set_identifier = (var.record_weight == null && var.record_failover == "") ? null : "${lower(var.usecase_name)}-${local.region}"
  dynamic "weighted_routing_policy" {
    for_each = var.record_weight == null ? {} : { weight_enabled = true }
    content {
      weight = var.record_weight
    }
  }
  dynamic "failover_routing_policy" {
    for_each = var.record_failover == "" ? {} : { fail_enabled = true }
    content {
      type = var.record_failover
    }
  }

  alias {
    evaluate_target_health = var.evaluate_target_health
    name                   = data.aws_vpc_endpoint.apigw-vpce.dns_entry[0].dns_name
    zone_id                = data.aws_vpc_endpoint.apigw-vpce.dns_entry[0].hosted_zone_id
  }
  health_check_id = var.health_check_id
}

data "aws_iam_policy_document" "domain_name_policy" {
  statement {
    effect = "Allow"
    principals {
      identifiers = ["*"]
      type        = "*"
    }
    actions = [
      "execute-api:Invoke"
    ]
    resources = [
      "*"
    ]
  }
  statement {
    effect = "Deny"
    principals {
      identifiers = ["*"]
      type        = "*"
    }
    actions = [
      "execute-api:Invoke"
    ]
    resources = [
      "*"
    ]
    condition {
      test     = "StringNotEquals"
      variable = "aws:SourceVpce"
      values = [
        data.aws_vpc_endpoint.apigw-vpce.id
      ]
    }
  }
}

resource "aws_api_gateway_domain_name" "custom-name" {
  domain_name     = local.fqdn
  certificate_arn = aws_acm_certificate_validation.cert-val.certificate_arn
  security_policy = "TLS_1_2"

  endpoint_configuration {
    types = ["PRIVATE"]
  }

  policy = data.aws_iam_policy_document.domain_name_policy.json

  tags = var.tags
}

resource "aws_api_gateway_domain_name_access_association" "assoc" {
  access_association_source      = data.aws_vpc_endpoint.apigw-vpce.id
  access_association_source_type = "VPCE"
  domain_name_arn                = aws_api_gateway_domain_name.custom-name.arn
}

resource "aws_api_gateway_base_path_mapping" "map" {
  count          = length(var.api_gateway_mappings)
  api_id         = var.api_gateway_mappings[count.index]["api_id"]
  domain_name    = aws_api_gateway_domain_name.custom-name.domain_name
  domain_name_id = aws_api_gateway_domain_name.custom-name.domain_name_id
  stage_name     = var.api_gateway_mappings[count.index]["stage_name"]
  base_path      = lookup(var.api_gateway_mappings[count.index], "base_path", null)
}

Steps to Reproduce

1. Do an initial apply to create the domain name with a policy
2. Change the policy
3. Do another apply and get the error.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ewbankkit]

@bkeech Thanks for raising this issue.
It has already been noticed in #40699. I'm going to close this one as a duplicate so that we can concentrate discussion in the linked issue.
Please add any additional comments there.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.