Destroying azurerm_managed_disks fail with "context deadline exceeded"

[ghost]

This issue was originally opened by @hantuzun as hashicorp/terraform#17302. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Destroying Azure VMs with managed disks (azurerm_managed_disk) hang until the deadline. The terraform apply command fails with StatusCode=204 -- Original Error: context deadline exceeded. Subsequent terraform apply commands succeed without any updates since the original command destroys the VM and its managed disk successfully.

I think the issue is due to an error on getting the response of managed disks state.

Terraform Version

Terraform v0.11.3
+ provider.azurerm v1.1.0
+ provider.external v1.0.0
+ provider.null v1.0.0
+ provider.random v1.1.0
+ provider.template v1.0.0

Terraform Configuration Files

https://github.com/dcos/terraform-dcos/tree/master/azure

Debug Output

azurerm_managed_disk.spark_agent_managed_disk.1: Still destroying... (ID: /subscriptions/de81f8e5-ae2f-43d7-ba87-...sks/sg-prod-north-europe-spark-agent-2, 59m40s elapsed)
2018/02/08 18:32:46 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/02/08 18:32:46 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
2018/02/08 18:32:46 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
2018/02/08 18:32:51 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
2018/02/08 18:32:51 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/02/08 18:32:51 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
azurerm_managed_disk.spark_agent_managed_disk.1: Still destroying... (ID: /subscriptions/de81f8e5-ae2f-43d7-ba87-...sks/sg-prod-north-europe-spark-agent-2, 59m50s elapsed)
2018/02/08 18:32:56 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/02/08 18:32:56 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
2018/02/08 18:32:56 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
2018/02/08 18:33:01 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
2018/02/08 18:33:01 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/02/08 18:33:01 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)"
azurerm_managed_disk.spark_agent_managed_disk.1: Still destroying... (ID: /subscriptions/de81f8e5-ae2f-43d7-ba87-...sks/sg-prod-north-europe-spark-agent-2, 1h0m0s elapsed)
2018/02/08 18:33:04 [TRACE] root: eval: *terraform.EvalWriteState
2018/02/08 18:33:04 [TRACE] root: eval: *terraform.EvalApplyPost
2018/02/08 18:33:04 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* azurerm_managed_disk.spark_agent_managed_disk.1: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded
2018/02/08 18:33:04 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* azurerm_managed_disk.spark_agent_managed_disk.1: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded
2018/02/08 18:33:04 [ERROR] root: eval: *terraform.EvalOpFilter, err: 1 error(s) occurred:

* azurerm_managed_disk.spark_agent_managed_disk.1: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded
2018/02/08 18:33:04 [TRACE] [walkApply] Exiting eval tree: azurerm_managed_disk.spark_agent_managed_disk[1] (destroy)
2018/02/08 18:33:04 [TRACE] dag/walk: upstream errored, not walking "provider.azurerm (close)"
2018/02/08 18:33:04 [TRACE] dag/walk: upstream errored, not walking "meta.count-boundary (count boundary fixup)"
2018/02/08 18:33:04 [TRACE] dag/walk: upstream errored, not walking "root"
2018/02/08 18:33:04 [TRACE] Preserving existing state lineage "56449d11-2b8c-4df9-bddc-bc480c9f78ca"
2018/02/08 18:33:04 [TRACE] Preserving existing state lineage "56449d11-2b8c-4df9-bddc-bc480c9f78ca"
2018/02/08 18:33:04 [TRACE] Preserving existing state lineage "56449d11-2b8c-4df9-bddc-bc480c9f78ca"
2018/02/08 18:33:04 [DEBUG] plugin: waiting for all plugin processes to complete...

Error: Error applying plan:

1 error(s) occurred:

* azurerm_managed_disk.spark_agent_managed_disk[1] (destroy): 1 error(s) occurred:

* azurerm_managed_disk.spark_agent_managed_disk.1: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.


2018-02-08T18:33:04.465Z [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2018/02/08 18:33:04 [DEBUG] plugin: waiting for all plugin processes to complete...
2018-02-08T18:33:04.465Z [DEBUG] plugin.terraform: file-provisioner (internal) 2018/02/08 18:33:04 [DEBUG] plugin: waiting for all plugin processes to complete...
2018-02-08T18:33:04.465Z [DEBUG] plugin: plugin process exited: path=/home/sg-prod/terraform-dcos/azure/.terraform/plugins/linux_amd64/terraform-provider-null_v1.0.0_x4
2018-02-08T18:33:04.465Z [DEBUG] plugin: plugin process exited: path=/home/sg-prod/terraform-dcos/azure/.terraform/plugins/linux_amd64/terraform-provider-azurerm_v1.1.0_x4
2018-02-08T18:33:04.465Z [DEBUG] plugin: plugin process exited: path=/home/sg-prod/terraform-dcos/azure/.terraform/plugins/linux_amd64/terraform-provider-random_v1.1.0_x4
2018-02-08T18:33:04.466Z [DEBUG] plugin: plugin process exited: path=/home/sg-prod/terraform-dcos/azure/.terraform/plugins/linux_amd64/terraform-provider-template_v1.0.0_x4
2018-02-08T18:33:04.466Z [DEBUG] plugin: plugin process exited: path=/usr/bin/terraform
2018-02-08T18:33:04.466Z [DEBUG] plugin: plugin process exited: path=/usr/bin/terraform
2018-02-08T18:33:04.466Z [DEBUG] plugin: plugin process exited: path=/home/sg-prod/terraform-dcos/azure/.terraform/plugins/linux_amd64/terraform-provider-external_v1.0.0_x4

Expected Behavior

Actual Behavior

Steps to Reproduce

• Clone https://github.com/dcos/terraform-dcos
• cd azure
• Create such a terraform.tfvars file:

cluster_name = "tf"
state = "install"
dcos_version = "1.10.2"
ssh_pub_key = "ssh-rsa ..."
os = "centos_7.3"
azure_region = "North Europe"
owner = "sg"
azure_admin_username = "sg-prod"
azure_master_instance_type = "Standard_D2_v3"
azure_agent_instance_type = "Standard_D11_v2_Promo"
azure_public_agent_instance_type = "Standard_F2"
azure_bootstrap_instance_type = "Standard_D1_v2"
num_of_masters = "3"
num_of_public_agents = "2"
num_of_private_agents = "1"
instance_disk_size = "128"
dcos_oauth_enabled = "false"
ssh_port = "2200"
dcos_process_timeout = "600"
dcos_telemetry_enabled = "false"
dcos_overlay_enable = "true"
dcos_use_proxy = "false"
dcos_check_time = "true"
dcos_docker_remove_delay = ""
dcos_audit_logging = "true"
dcos_enable_docker_gc = "true"

• terraform apply
• Update num_of_public_agents = "2" to num_of_public_agents = 1"
• terraform apply

Additional Context

Nothing is atypical as far as I know. I'm running Terraform on new Centos 7.4 machines.

[hantuzun]

I thought this issue could be about terraform but here we are.

[schmandforke]

got an issue here with the latest terraform:light container and "some" disks:

Error: Error applying plan:

28 error(s) occurred:

* azurerm_managed_disk.wf_disk_page[0] (destroy): 1 error(s) occurred:

* azurerm_managed_disk.wf_disk_page.0: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded
* azurerm_managed_disk.app_disk_log[0] (destroy): 1 error(s) occurred:

* azurerm_managed_disk.app_disk_log.0: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded
* azurerm_managed_disk.app_disk_software[0] (destroy): 1 error(s) occurred:

* azurerm_managed_disk.app_disk_software.0: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded
* azurerm_managed_disk.web_disk_log[1] (destroy): 1 error(s) occurred:

* azurerm_managed_disk.web_disk_log.1: azure#WaitForCompletion: context has been cancelled: StatusCode=204 -- Original Error: context deadline exceeded
* azurerm_managed_disk.wf_disk_page[1] (destroy): 1 error(s) occurred:

...

[hantuzun]

We're constantly running into this issue while destroying various types of machines.

[schmandforke]

there is an issue with nearly the same content:
#171 (comment)

would be great if the azurerm provider would get a timeout parameter...

[seanknox]

I just hit this with Terraform v0.11.3 and latest version of the Azure provider (as of 20180325). Created a VM with managed disk with delete_os_disk_on_termination and delete_data_disks_on_termination set to true. The disks are shown as deleted in the Azure Portal, but the (Terraform) provider ultimately fails.

[tombuildsstuff]

Closing this in favour of #171, where this will be fixed in the near future

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!