Deleting an aws.ec2.Vpc associated with an IPAM Pool can take 25min

[t0yv0]

Describe what happened

It appears that deleting an aws.ec2.Vpc resource can take a long time if this resource was tied to an IPAM pool.

This behavior is inherited from upstream. IN hashicorp/terraform-provider-aws#30795 specifically commit 563fe2548e7bb32060aa0d91a1be91f6cdb65655 code is introduced to wait for IPAM to remove allocations before considering the VPC to be deleted. This can take a long time. The original timeout was set at 20min. IT has since been updated to 30min.

Perhaps more surprisingly, setting an explicit customTimeout in Pulumi is not helping to short-circuit the deletion.

Sample program

Using this program:

import * as aws from "@pulumi/aws";

const vpc = new aws.ec2.Vpc("my-vpc", {
  ipv4IpamPoolId: "ipam-pool-08a759a200748ae22",
  ipv4NetmaskLength: 24,
}, {
  customTimeouts: {
    delete: "1m",
  },
});

export const vpcId = vpc.id;

IPAM was configured with:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const config = new pulumi.Config();

const operation: string = config.get("operation") || "create";

const repository = "pulumi/pulumi-awsx";

const tags = {
  "repository": repository,
};

const currentRegion = aws.getRegionOutput({});

export let vpcIpamPoolId: pulumi.Output<string>;

const vpcIpamPoolDescription = `IPAM Pool for ${repository} testing`;

if (operation === "create") {
  const myVpcIpam = new aws.ec2.VpcIpam("myVpcIpam", {
    operatingRegions: [{
      regionName: currentRegion.name,
    }],
    description: `IPAM for ${repository} testing`,
    tags: tags,
  });

  const myVpcIpamPool = new aws.ec2.VpcIpamPool("myVpcIpamPool", {
    addressFamily: "ipv4",
    ipamScopeId: myVpcIpam.privateDefaultScopeId,
    description: vpcIpamPoolDescription,
    locale: currentRegion.name,
    tags: tags,
  });

  new aws.ec2.VpcIpamPoolCidr("myVpcIpamPoolCidr", {
    ipamPoolId: myVpcIpamPool.id,
    cidr: "172.20.0.0/16",
  });

  vpcIpamPoolId = myVpcIpamPool.id;

} else {
  const myVpcIpamPoolRef = aws.ec2.getVpcIpamPool({
    filters: [
      {
        name: "description",
        values: [vpcIpamPoolDescription],
      },
    ]
  }).then(v => v.id||"").catch(() => "");

  vpcIpamPoolId = pulumi.output(myVpcIpamPoolRef);
}

Log output

The relevant bits are:

I0805 15:02:32.046580   56309 eventsink.go:62] eventSink::Debug(<{%reset%}>  logging/tf_logger.go:45: provider: HTTP Response Received: [email protected] tf_aws.sdk=aws-sdk-go-v2 rpc.service=EC2 http.response.header.strict_transport_security="max-age=31536000; includeSubDomains" http.duration=951 aws.region=us-east-1 http.response.header.x_amzn_requestid=52a556ce-e7ab-4fc5-b092-9f5e0d85942a http.response.header.vary=accept-encoding http.response.header.cache_control="no-cache, no-store" http.status_code=200
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <GetIpamPoolAllocationsResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
  |     <requestId>52a556ce-e7ab-4fc5-b092-9f5e0d85942a</requestId>
  |     <ipamPoolAllocationSet>
  |         <item>
  |             <allocationId>ipam-pool-alloc-0d47c1f6674f54e51aae14beedcf9e295</allocationId>
  |             <cidr>172.20.0.0/24</cidr>
  |             <cidrBlock>172.20.0.0/24</cidrBlock>
  |             <ipamPoolAllocationId>ipam-pool-alloc-0d47c1f6674f54e51aae14beedcf9e295</ipamPoolAllocationId>
  |             <resourceId>vpc-0c4d8e11e79b1f212</resourceId>
  |             <resourceOwner>616138583583</resourceOwner>
  |             <resourceRegion>us-east-1</resourceRegion>
  |             <resourceType>vpc</resourceType>
  |         </item>
  |         <item>
  |             <allocationId>ipam-pool-alloc-0d23770bd2054408a9124674b91e84454</allocationId>
  |             <cidr>172.20.1.0/24</cidr>
  |             <cidrBlock>172.20.1.0/24</cidrBlock>
  |             <ipamPoolAllocationId>ipam-pool-alloc-0d23770bd2054408a9124674b91e84454</ipamPoolAllocationId>
  |             <resourceId>vpc-053ef2cb43494b838</resourceId>
  |             <resourceOwner>616138583583</resourceOwner>
  |             <resourceRegion>us-east-1</resourceRegion>
  |             <resourceType>vpc</resourceType>
  |         </item>
  |         <item>
  |             <allocationId>ipam-pool-alloc-00c2a3884ca7c462fa4bd537792b14883</allocationId>
  |             <cidr>172.20.2.0/24</cidr>
  |             <cidrBlock>172.20.2.0/24</cidrBlock>
  |             <ipamPoolAllocationId>ipam-pool-alloc-00c2a3884ca7c462fa4bd537792b14883</ipamPoolAllocationId>
  |             <resourceId>vpc-0397981d998d55bff</resourceId>
  |             <resourceOwner>616138583583</resourceOwner>
  |             <resourceRegion>us-east-1</resourceRegion>
  |             <resourceType>vpc</resourceType>
  |         </item>
  |     </ipamPoolAllocationSet>
  | </GetIpamPoolAllocationsResponse>
   rpc.method=GetIpamPoolAllocations rpc.system=aws-api http.response.header.date="Mon, 05 Aug 2024 20:02:31 GMT" http.response.header.server=AmazonEC2 tf_aws.signing_region="" http.response.header.content_type=text/xml;charset=UTF-8
<{%reset%}>)
I0805 15:02:32.047707   56309 eventsink.go:59] [TRACE] Waiting 10s before next try
I0805 15:02:32.047793   56309 eventsink.go:62] eventSink::Debug(<{%reset%}>[TRACE] Waiting 10s before next try<{%re```

Affected Resource(s)

aws.ec2.Vpc

Output of pulumi about

CLI          
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  aws     6.48.0
resource  awsx    2.13.0
resource  docker  4.5.5
resource  docker  3.6.1
language  nodejs  unknown

Host     
OS       darwin
Version  14.5
Arch     arm64

This project is written in nodejs: executable='/Users/anton/bin/node' version='v18.18.2'

Current Stack: anton-pulumi-corp/slow-vpc-delete/dev

Found no resources associated with dev

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/anton-pulumi-corp
User           anton-pulumi-corp
Organizations  anton-pulumi-corp, moolumi, demo, pulumi
Token type     personal

Dependencies:
NAME            VERSION
@types/node     18.19.43
typescript      5.5.4
@pulumi/aws     6.48.0
@pulumi/awsx    2.13.0
@pulumi/pulumi  3.128.0

Pulumi locates its logs in /var/folders/gd/3ncjb1lj5ljgk8xl5ssn_gvc0000gn/T/com.apple.shortcuts.mac-helper// by default

Additional context

N/A

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Surprisingly terminating the pulumi destroy and trying again seems to succeed in deleting the VPC in this case, as a workaround.