Possibility to change update behavior for ibm_container_cluster

[jkayani]

Hi,

We have a use case of creating and updating IBM Cloud IKS/ROKS clusters via Terraform. We execute terraform apply in our CI/CD system, which is subject to time limits on jobs.

We noticed 2 things with the update_all_workers argument:

• It only tries to update workers if the same terraform apply registers a change in the master version: https://github.com/IBM-Cloud/terraform-provider-ibm/blob/master/ibm/resource_ibm_container_cluster.go#L818. This can be a problem in the case where the master update finishes, but the terraform apply fails shortly after. We had this problem recently, and were unable to complete the cluster upgrade (master and workers) purely via Terraform.

• It waits for each worker to finish before continuing to the next one: https://github.com/IBM-Cloud/terraform-provider-ibm/blob/master/ibm/resource_ibm_container_cluster.go#L858. In our case, we had clusters of 9 nodes - this of course caused our CI/CD system to terminate terraform apply before it finished.

For the first observation, we were wondering if it'd be possible to modify the logic so that if update_all_workers is true, and the master version doesn't match the worker version, to plan worker updates.

For the second observation, it makes sense why it's done that way, since Terraform has to wait for the operation to complete before recording success in the state. However, we were wondering if we could have an option like wait_for_workers that we could set false for our use case. Then, we could trigger worker updates (ensuring each worker received the update command), have the CI/CD job finish on time, and simply terraform refresh the state of the cluster once the update is complete.

[TBradCreech]

Subscribing to this issue as a VERY interested party working in IBM Watson Health. Another common scenario that should not require a master update, is occasions where there is only a worker update available (usually happens every two weeks) and no master update. So the desire is to update the workers while within the context of the same existing major.minor, where no change occurs to the master.

It is also worth noting that our observation when using "update _all_workers", is all workers updated at the same time -- which is contrary to what Josh reported when opening this issue. Perhaps that is because the workers were of provider type "vpc-gen2".

Obviously, updating all workers at once would result in a down system which is not tolerable for our (most) cloud solutions.

Josh @jkayani , please confirm that your result was using classic or vpc-classic, and not vpc-gen2 -- which would probablyt explain why we see different behavior of all workers being done at once vs. one-at-a-time.

[werne2j]

@TBradCreech we are using classic clusters. The reason that we see saw a one at a time versus all at once appears to be the way the code was written.

If you look at the code here https://github.com/IBM-Cloud/terraform-provider-ibm/blob/master/ibm/resource_ibm_container_cluster.go#L849 you can see looping over each worker and updating then waiting for the update to finish, before moving on to the next worker.

Looking at the vpc code it does not appear to "wait" for each worker to finish https://github.com/IBM-Cloud/terraform-provider-ibm/blob/master/ibm/resource_ibm_container_vpc_cluster.go#L574.

[jkayani]

@TBradCreech I confirm that yes, our use case was regarding classic Openshift 4 clusters on IBM Cloud. No VPC at all right now.

[Anil-CM]

@jkayani we addressed these issues in the PR - #1989

The PR addresses the following:

1. Kube version of nodes updated serially
2. Separated the dependency of worker nodes kube version upgradation. - To update only the worker nodes update_all_workers.
3. New parameter - wait_for_worker_update is introduced. To avoid more time in updating kube version of worker nodes.
   Note : This will cause cluster downtime.

we are coming up with a new resource for kube version upgrade to handle all the requirements.

[hkantare]

Fixed in latest release
https://github.com/IBM-Cloud/terraform-provider-ibm/releases/tag/v1.14.0

[tpolekhin]

@hkantare I was testing this feature for last 2 days and I wasn't able to make it work

$ terraform version
Terraform v0.12.29
+ provider.ibm v1.14.0

Terraform code

provider "ibm" {
  generation = 2
}

data "ibm_resource_group" "group" {
  name = "Default"
}

resource "ibm_is_vpc" "vpc" {
  name                      = "icm-poc-vpc"
  address_prefix_management = "manual"
  resource_group            = data.ibm_resource_group.group.id
}

###############

resource "ibm_is_vpc_address_prefix" "us-south-1" {
  name = "icm-poc-vpc-prefix-us-south-1"
  zone = "us-south-1"
  vpc  = ibm_is_vpc.vpc.id
  cidr = "10.10.0.0/16"
}

resource "ibm_is_vpc_address_prefix" "us-south-2" {
  name = "icm-poc-vpc-prefix-us-south-2"
  zone = "us-south-2"
  vpc  = ibm_is_vpc.vpc.id
  cidr = "10.20.0.0/16"
}

resource "ibm_is_vpc_address_prefix" "us-south-3" {
  name = "icm-poc-vpc-prefix-us-south-3"
  zone = "us-south-3"
  vpc  = ibm_is_vpc.vpc.id
  cidr = "10.30.0.0/16"
}

#############

resource "ibm_is_subnet" "us-south-1" {
  name            = "icm-poc-vpc-us-south-1-subnet-1"
  vpc             = ibm_is_vpc.vpc.id
  zone            = "us-south-1"
  public_gateway  = ibm_is_public_gateway.us-south-1.id
  ipv4_cidr_block = "10.10.0.0/24"
  depends_on      = [ibm_is_vpc_address_prefix.us-south-1]
}

resource "ibm_is_subnet" "us-south-2" {
  name            = "icm-poc-vpc-us-south-2-subnet-1"
  vpc             = ibm_is_vpc.vpc.id
  zone            = "us-south-2"
  public_gateway  = ibm_is_public_gateway.us-south-2.id
  ipv4_cidr_block = "10.20.0.0/24"
  depends_on      = [ibm_is_vpc_address_prefix.us-south-2]
}

resource "ibm_is_subnet" "us-south-3" {
  name            = "icm-poc-vpc-us-south-3-subnet-1"
  vpc             = ibm_is_vpc.vpc.id
  zone            = "us-south-3"
  public_gateway  = ibm_is_public_gateway.us-south-3.id
  ipv4_cidr_block = "10.30.0.0/24"
  depends_on      = [ibm_is_vpc_address_prefix.us-south-3]
}

###############

resource "ibm_is_public_gateway" "us-south-1" {
  name = "icm-poc-vpc-gateway-us-south-1"
  vpc  = ibm_is_vpc.vpc.id
  zone = "us-south-1"
}

resource "ibm_is_public_gateway" "us-south-2" {
  name = "icm-poc-vpc-gateway-us-south-2"
  vpc  = ibm_is_vpc.vpc.id
  zone = "us-south-2"
}

resource "ibm_is_public_gateway" "us-south-3" {
  name = "icm-poc-vpc-gateway-us-south-3"
  vpc  = ibm_is_vpc.vpc.id
  zone = "us-south-3"
}

###############

resource "ibm_is_security_group_rule" "allow-lb-us-south-1-subnet-1" {
  group     = ibm_is_vpc.vpc.default_security_group
  direction = "inbound"
  remote    = "10.10.0.0/24"
  tcp {
    port_min = 30000
    port_max = 32767
  }
}

resource "ibm_is_security_group_rule" "allow-lb-us-south-2-subnet-1" {
  group     = ibm_is_vpc.vpc.default_security_group
  direction = "inbound"
  remote    = "10.20.0.0/24"
  tcp {
    port_min = 30000
    port_max = 32767
  }
}

resource "ibm_is_security_group_rule" "allow-lb-us-south-3-subnet-1" {
  group     = ibm_is_vpc.vpc.default_security_group
  direction = "inbound"
  remote    = "10.30.0.0/24"
  tcp {
    port_min = 30000
    port_max = 32767
  }
}

resource "ibm_container_vpc_cluster" "cluster" {
  name                   = "icm-poc-vpc-cluster"
  vpc_id                 = ibm_is_vpc.vpc.id
  flavor                 = "bx2.4x16"
  worker_count           = "1"
  resource_group_id      = data.ibm_resource_group.group.id
  kube_version           = "1.18"
  pod_subnet             = "172.21.0.0/16"
  service_subnet         = "172.31.0.0/16"
  update_all_workers     = true
  wait_for_worker_update = true

  worker_labels = {
    type = "general"
  }
  lifecycle {
    ignore_changes = [worker_labels]
  }
  zones {
    subnet_id = ibm_is_subnet.us-south-1.id
    name      = "us-south-1"
  }
  zones {
    subnet_id = ibm_is_subnet.us-south-2.id
    name      = "us-south-2"
  }
  zones {
    subnet_id = ibm_is_subnet.us-south-3.id
    name      = "us-south-3"
  }
}

resource "ibm_container_vpc_worker_pool" "cassandra" {
  cluster           = ibm_container_vpc_cluster.cluster.id
  worker_pool_name  = "cassandra"
  flavor            = "cx2.2x4"
  vpc_id            = ibm_is_vpc.vpc.id
  worker_count      = "1"
  resource_group_id = data.ibm_resource_group.group.id

  zones {
    subnet_id = ibm_is_subnet.us-south-1.id
    name      = "us-south-1"
  }
  zones {
    subnet_id = ibm_is_subnet.us-south-2.id
    name      = "us-south-2"
  }
  zones {
    subnet_id = ibm_is_subnet.us-south-3.id
    name      = "us-south-3"
  }
}

Initial deployment

data.ibm_resource_group.group: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # ibm_container_vpc_cluster.cluster will be created
  + resource "ibm_container_vpc_cluster" "cluster" {
      + albs                            = (known after apply)
      + crn                             = (known after apply)
      + disable_public_service_endpoint = false
      + flavor                          = "bx2.4x16"
      + force_delete_storage            = false
      + id                              = (known after apply)
      + ingress_hostname                = (known after apply)
      + ingress_secret                  = (sensitive value)
      + kube_version                    = "1.18"
      + master_status                   = (known after apply)
      + master_url                      = (known after apply)
      + name                            = "icm-poc-vpc-cluster"
      + pod_subnet                      = "172.21.0.0/16"
      + private_service_endpoint_url    = (known after apply)
      + public_service_endpoint_url     = (known after apply)
      + resource_controller_url         = (known after apply)
      + resource_crn                    = (known after apply)
      + resource_group_id               = "af3e3dfffaf444579188b817bb7b320e"
      + resource_group_name             = (known after apply)
      + resource_name                   = (known after apply)
      + resource_status                 = (known after apply)
      + service_subnet                  = "172.31.0.0/16"
      + state                           = (known after apply)
      + tags                            = (known after apply)
      + update_all_workers              = true
      + vpc_id                          = (known after apply)
      + wait_for_worker_update          = true
      + wait_till                       = "IngressReady"
      + worker_count                    = 1
      + worker_labels                   = {
          + "type" = "general"
        }

      + zones {
          + name      = "us-south-1"
          + subnet_id = (known after apply)
        }
      + zones {
          + name      = "us-south-2"
          + subnet_id = (known after apply)
        }
      + zones {
          + name      = "us-south-3"
          + subnet_id = (known after apply)
        }
    }

  # ibm_container_vpc_worker_pool.cassandra will be created
  + resource "ibm_container_vpc_worker_pool" "cassandra" {
      + cluster           = (known after apply)
      + flavor            = "cx2.2x4"
      + id                = (known after apply)
      + resource_group_id = "af3e3dfffaf444579188b817bb7b320e"
      + vpc_id            = (known after apply)
      + worker_count      = 1
      + worker_pool_name  = "cassandra"

      + zones {
          + name      = "us-south-1"
          + subnet_id = (known after apply)
        }
      + zones {
          + name      = "us-south-2"
          + subnet_id = (known after apply)
        }
      + zones {
          + name      = "us-south-3"
          + subnet_id = (known after apply)
        }
    }

  # ibm_is_public_gateway.us-south-1 will be created
  + resource "ibm_is_public_gateway" "us-south-1" {
      + floating_ip             = (known after apply)
      + id                      = (known after apply)
      + name                    = "icm-poc-vpc-gateway-us-south-1"
      + resource_controller_url = (known after apply)
      + resource_crn            = (known after apply)
      + resource_group          = (known after apply)
      + resource_group_name     = (known after apply)
      + resource_name           = (known after apply)
      + resource_status         = (known after apply)
      + status                  = (known after apply)
      + tags                    = (known after apply)
      + vpc                     = (known after apply)
      + zone                    = "us-south-1"
    }

  # ibm_is_public_gateway.us-south-2 will be created
  + resource "ibm_is_public_gateway" "us-south-2" {
      + floating_ip             = (known after apply)
      + id                      = (known after apply)
      + name                    = "icm-poc-vpc-gateway-us-south-2"
      + resource_controller_url = (known after apply)
      + resource_crn            = (known after apply)
      + resource_group          = (known after apply)
      + resource_group_name     = (known after apply)
      + resource_name           = (known after apply)
      + resource_status         = (known after apply)
      + status                  = (known after apply)
      + tags                    = (known after apply)
      + vpc                     = (known after apply)
      + zone                    = "us-south-2"
    }

  # ibm_is_public_gateway.us-south-3 will be created
  + resource "ibm_is_public_gateway" "us-south-3" {
      + floating_ip             = (known after apply)
      + id                      = (known after apply)
      + name                    = "icm-poc-vpc-gateway-us-south-3"
      + resource_controller_url = (known after apply)
      + resource_crn            = (known after apply)
      + resource_group          = (known after apply)
      + resource_group_name     = (known after apply)
      + resource_name           = (known after apply)
      + resource_status         = (known after apply)
      + status                  = (known after apply)
      + tags                    = (known after apply)
      + vpc                     = (known after apply)
      + zone                    = "us-south-3"
    }

  # ibm_is_security_group_rule.allow-lb-us-south-1-subnet-1 will be created
  + resource "ibm_is_security_group_rule" "allow-lb-us-south-1-subnet-1" {
      + direction   = "inbound"
      + group       = (known after apply)
      + id          = (known after apply)
      + ip_version  = "ipv4"
      + related_crn = (known after apply)
      + remote      = "10.10.0.0/24"
      + rule_id     = (known after apply)

      + tcp {
          + port_max = 32767
          + port_min = 30000
        }
    }

  # ibm_is_security_group_rule.allow-lb-us-south-2-subnet-1 will be created
  + resource "ibm_is_security_group_rule" "allow-lb-us-south-2-subnet-1" {
      + direction   = "inbound"
      + group       = (known after apply)
      + id          = (known after apply)
      + ip_version  = "ipv4"
      + related_crn = (known after apply)
      + remote      = "10.20.0.0/24"
      + rule_id     = (known after apply)

      + tcp {
          + port_max = 32767
          + port_min = 30000
        }
    }

  # ibm_is_security_group_rule.allow-lb-us-south-3-subnet-1 will be created
  + resource "ibm_is_security_group_rule" "allow-lb-us-south-3-subnet-1" {
      + direction   = "inbound"
      + group       = (known after apply)
      + id          = (known after apply)
      + ip_version  = "ipv4"
      + related_crn = (known after apply)
      + remote      = "10.30.0.0/24"
      + rule_id     = (known after apply)

      + tcp {
          + port_max = 32767
          + port_min = 30000
        }
    }

  # ibm_is_subnet.us-south-1 will be created
  + resource "ibm_is_subnet" "us-south-1" {
      + available_ipv4_address_count = (known after apply)
      + id                           = (known after apply)
      + ipv4_cidr_block              = "10.10.0.0/24"
      + ipv6_cidr_block              = (known after apply)
      + name                         = "icm-poc-vpc-us-south-1-subnet-1"
      + network_acl                  = (known after apply)
      + public_gateway               = (known after apply)
      + resource_controller_url      = (known after apply)
      + resource_crn                 = (known after apply)
      + resource_group               = (known after apply)
      + resource_group_name          = (known after apply)
      + resource_name                = (known after apply)
      + resource_status              = (known after apply)
      + status                       = (known after apply)
      + total_ipv4_address_count     = (known after apply)
      + vpc                          = (known after apply)
      + zone                         = "us-south-1"
    }

  # ibm_is_subnet.us-south-2 will be created
  + resource "ibm_is_subnet" "us-south-2" {
      + available_ipv4_address_count = (known after apply)
      + id                           = (known after apply)
      + ipv4_cidr_block              = "10.20.0.0/24"
      + ipv6_cidr_block              = (known after apply)
      + name                         = "icm-poc-vpc-us-south-2-subnet-1"
      + network_acl                  = (known after apply)
      + public_gateway               = (known after apply)
      + resource_controller_url      = (known after apply)
      + resource_crn                 = (known after apply)
      + resource_group               = (known after apply)
      + resource_group_name          = (known after apply)
      + resource_name                = (known after apply)
      + resource_status              = (known after apply)
      + status                       = (known after apply)
      + total_ipv4_address_count     = (known after apply)
      + vpc                          = (known after apply)
      + zone                         = "us-south-2"
    }

  # ibm_is_subnet.us-south-3 will be created
  + resource "ibm_is_subnet" "us-south-3" {
      + available_ipv4_address_count = (known after apply)
      + id                           = (known after apply)
      + ipv4_cidr_block              = "10.30.0.0/24"
      + ipv6_cidr_block              = (known after apply)
      + name                         = "icm-poc-vpc-us-south-3-subnet-1"
      + network_acl                  = (known after apply)
      + public_gateway               = (known after apply)
      + resource_controller_url      = (known after apply)
      + resource_crn                 = (known after apply)
      + resource_group               = (known after apply)
      + resource_group_name          = (known after apply)
      + resource_name                = (known after apply)
      + resource_status              = (known after apply)
      + status                       = (known after apply)
      + total_ipv4_address_count     = (known after apply)
      + vpc                          = (known after apply)
      + zone                         = "us-south-3"
    }

  # ibm_is_vpc.vpc will be created
  + resource "ibm_is_vpc" "vpc" {
      + address_prefix_management = "manual"
      + classic_access            = false
      + crn                       = (known after apply)
      + cse_source_addresses      = (known after apply)
      + default_network_acl       = (known after apply)
      + default_security_group    = (known after apply)
      + id                        = (known after apply)
      + name                      = "icm-poc-vpc"
      + resource_controller_url   = (known after apply)
      + resource_crn              = (known after apply)
      + resource_group            = "af3e3dfffaf444579188b817bb7b320e"
      + resource_group_name       = (known after apply)
      + resource_name             = (known after apply)
      + resource_status           = (known after apply)
      + security_group            = (known after apply)
      + status                    = (known after apply)
      + subnets                   = (known after apply)
      + tags                      = (known after apply)
    }

  # ibm_is_vpc_address_prefix.us-south-1 will be created
  + resource "ibm_is_vpc_address_prefix" "us-south-1" {
      + cidr        = "10.10.0.0/16"
      + has_subnets = (known after apply)
      + id          = (known after apply)
      + name        = "icm-poc-vpc-prefix-us-south-1"
      + related_crn = (known after apply)
      + vpc         = (known after apply)
      + zone        = "us-south-1"
    }

  # ibm_is_vpc_address_prefix.us-south-2 will be created
  + resource "ibm_is_vpc_address_prefix" "us-south-2" {
      + cidr        = "10.20.0.0/16"
      + has_subnets = (known after apply)
      + id          = (known after apply)
      + name        = "icm-poc-vpc-prefix-us-south-2"
      + related_crn = (known after apply)
      + vpc         = (known after apply)
      + zone        = "us-south-2"
    }

  # ibm_is_vpc_address_prefix.us-south-3 will be created
  + resource "ibm_is_vpc_address_prefix" "us-south-3" {
      + cidr        = "10.30.0.0/16"
      + has_subnets = (known after apply)
      + id          = (known after apply)
      + name        = "icm-poc-vpc-prefix-us-south-3"
      + related_crn = (known after apply)
      + vpc         = (known after apply)
      + zone        = "us-south-3"
    }

Plan: 15 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

ibm_is_vpc.vpc: Creating...
ibm_is_vpc.vpc: Still creating... [10s elapsed]
ibm_is_vpc.vpc: Creation complete after 18s [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30]
ibm_is_vpc_address_prefix.us-south-2: Creating...
ibm_is_vpc_address_prefix.us-south-1: Creating...
ibm_is_vpc_address_prefix.us-south-3: Creating...
ibm_is_security_group_rule.allow-lb-us-south-2-subnet-1: Creating...
ibm_is_security_group_rule.allow-lb-us-south-1-subnet-1: Creating...
ibm_is_security_group_rule.allow-lb-us-south-3-subnet-1: Creating...
ibm_is_public_gateway.us-south-3: Creating...
ibm_is_public_gateway.us-south-2: Creating...
ibm_is_public_gateway.us-south-1: Creating...
ibm_is_vpc_address_prefix.us-south-1: Creation complete after 1s [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30/r006-ab02b5c2-fa61-4cac-a824-887aa3996c31]
ibm_is_security_group_rule.allow-lb-us-south-2-subnet-1: Creation complete after 2s [id=r006-83afafd9-713b-4f6f-b275-85913db5bbfc.r006-b0115af0-de01-4b5d-8f3f-7beac5f14dfc]
ibm_is_security_group_rule.allow-lb-us-south-1-subnet-1: Creation complete after 2s [id=r006-83afafd9-713b-4f6f-b275-85913db5bbfc.r006-bb54a95e-a459-48f8-8491-357426687c44]
ibm_is_vpc_address_prefix.us-south-2: Creation complete after 2s [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30/r006-a4229990-e3e9-49f6-be64-8fbcf39f396b]
ibm_is_security_group_rule.allow-lb-us-south-3-subnet-1: Creation complete after 3s [id=r006-83afafd9-713b-4f6f-b275-85913db5bbfc.r006-e1abd706-ee7d-4347-aa95-b51419fc0673]
ibm_is_vpc_address_prefix.us-south-3: Creation complete after 3s [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30/r006-4fa130e0-a3b8-417d-9c3b-262843cc227a]
ibm_is_public_gateway.us-south-3: Still creating... [10s elapsed]
ibm_is_public_gateway.us-south-1: Still creating... [10s elapsed]
ibm_is_public_gateway.us-south-2: Still creating... [10s elapsed]
ibm_is_public_gateway.us-south-2: Creation complete after 12s [id=r006-9ae859f4-5c15-4f86-a6eb-b632b2e359c5]
ibm_is_subnet.us-south-2: Creating...
ibm_is_public_gateway.us-south-1: Creation complete after 13s [id=r006-abf96eca-2def-4ecb-a913-99fa2005b0a3]
ibm_is_subnet.us-south-1: Creating...
ibm_is_public_gateway.us-south-3: Creation complete after 13s [id=r006-62424e43-b7fe-4f5e-836b-cd20955ea53a]
ibm_is_subnet.us-south-3: Creating...
ibm_is_subnet.us-south-2: Still creating... [10s elapsed]
ibm_is_subnet.us-south-1: Still creating... [10s elapsed]
ibm_is_subnet.us-south-3: Still creating... [10s elapsed]
ibm_is_subnet.us-south-1: Creation complete after 11s [id=0717-9623f8e0-82f6-440c-be17-593e9ec5f02e]
ibm_is_subnet.us-south-2: Creation complete after 12s [id=0727-54f8f9b8-6a11-448e-a4f4-7183f6ab6878]
ibm_is_subnet.us-south-3: Creation complete after 12s [id=0737-cf15cf67-02c4-4974-bb07-812a6ece71e0]
ibm_container_vpc_cluster.cluster: Creating...
ibm_container_vpc_cluster.cluster: Still creating... [10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [1m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [1m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [1m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [1m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [1m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [1m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [2m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [2m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [2m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [2m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [2m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [2m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [3m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [3m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [3m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [3m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [3m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [3m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [4m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [4m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [4m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [4m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [4m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [4m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [5m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [5m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [5m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [5m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [5m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [5m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [6m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [6m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [6m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [6m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [6m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [6m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [7m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [7m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [7m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [7m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [7m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [7m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [8m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [8m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [8m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [8m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [8m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [8m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [9m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [9m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [9m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [9m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [9m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [9m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [10m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [10m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [10m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [10m30s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [10m40s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [10m50s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [11m0s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [11m10s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [11m20s elapsed]
ibm_container_vpc_cluster.cluster: Still creating... [11m30s elapsed]
ibm_container_vpc_cluster.cluster: Creation complete after 11m33s [id=buluub2d0sl8ojim19kg]
ibm_container_vpc_worker_pool.cassandra: Creating...
ibm_container_vpc_worker_pool.cassandra: Still creating... [10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [1m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [1m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [1m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [1m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [1m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [1m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [2m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [2m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [2m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [2m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [2m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [2m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [3m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [3m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [3m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [3m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [3m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [3m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [4m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [4m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [4m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [4m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [4m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [4m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [5m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [5m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [5m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [5m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [5m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [5m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [6m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [6m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [6m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [6m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [6m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [6m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [7m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [7m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [7m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [7m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [7m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [7m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [8m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [8m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [8m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [8m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [8m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [8m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [9m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [9m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [9m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [9m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [9m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [9m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [10m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [10m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [10m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [10m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [10m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [10m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [11m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [11m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [11m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [11m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [11m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [11m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [12m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [12m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [12m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [12m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [12m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [12m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [13m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [13m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [13m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [13m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [13m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [13m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [14m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [14m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [14m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [14m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [14m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [14m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [15m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [15m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [15m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [15m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [15m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [15m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [16m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [16m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [16m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [16m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [16m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [16m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [17m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [17m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [17m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [17m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [17m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [17m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [18m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [18m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [18m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [18m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [18m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [18m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [19m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [19m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [19m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [19m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [19m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [19m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [20m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [20m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [20m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [20m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [20m40s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [20m50s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [21m0s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [21m10s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [21m20s elapsed]
ibm_container_vpc_worker_pool.cassandra: Still creating... [21m30s elapsed]
ibm_container_vpc_worker_pool.cassandra: Creation complete after 21m34s [id=buluub2d0sl8ojim19kg/buluub2d0sl8ojim19kg-788a2eb]

Apply complete! Resources: 15 added, 0 changed, 0 destroyed.

Updated ibm_container_vpc_cluster.kube_version to 1.19

Second deployment

data.ibm_resource_group.group: Refreshing state...
ibm_is_vpc.vpc: Refreshing state... [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30]
ibm_is_public_gateway.us-south-2: Refreshing state... [id=r006-9ae859f4-5c15-4f86-a6eb-b632b2e359c5]
ibm_is_security_group_rule.allow-lb-us-south-3-subnet-1: Refreshing state... [id=r006-83afafd9-713b-4f6f-b275-85913db5bbfc.r006-e1abd706-ee7d-4347-aa95-b51419fc0673]
ibm_is_security_group_rule.allow-lb-us-south-1-subnet-1: Refreshing state... [id=r006-83afafd9-713b-4f6f-b275-85913db5bbfc.r006-bb54a95e-a459-48f8-8491-357426687c44]
ibm_is_security_group_rule.allow-lb-us-south-2-subnet-1: Refreshing state... [id=r006-83afafd9-713b-4f6f-b275-85913db5bbfc.r006-b0115af0-de01-4b5d-8f3f-7beac5f14dfc]
ibm_is_vpc_address_prefix.us-south-2: Refreshing state... [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30/r006-a4229990-e3e9-49f6-be64-8fbcf39f396b]
ibm_is_vpc_address_prefix.us-south-3: Refreshing state... [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30/r006-4fa130e0-a3b8-417d-9c3b-262843cc227a]
ibm_is_public_gateway.us-south-1: Refreshing state... [id=r006-abf96eca-2def-4ecb-a913-99fa2005b0a3]
ibm_is_vpc_address_prefix.us-south-1: Refreshing state... [id=r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30/r006-ab02b5c2-fa61-4cac-a824-887aa3996c31]
ibm_is_public_gateway.us-south-3: Refreshing state... [id=r006-62424e43-b7fe-4f5e-836b-cd20955ea53a]
ibm_is_subnet.us-south-2: Refreshing state... [id=0727-54f8f9b8-6a11-448e-a4f4-7183f6ab6878]
ibm_is_subnet.us-south-1: Refreshing state... [id=0717-9623f8e0-82f6-440c-be17-593e9ec5f02e]
ibm_is_subnet.us-south-3: Refreshing state... [id=0737-cf15cf67-02c4-4974-bb07-812a6ece71e0]
ibm_container_vpc_cluster.cluster: Refreshing state... [id=bumgbamd0b6q6275pte0]
ibm_container_vpc_worker_pool.cassandra: Refreshing state... [id=bumgbamd0b6q6275pte0/bumgbamd0b6q6275pte0-e368d26]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # ibm_container_vpc_cluster.cluster will be updated in-place
  ~ resource "ibm_container_vpc_cluster" "cluster" {
        albs                            = [
            {
                alb_type               = "private"
                disable_deployment     = false
                enable                 = false
                id                     = "private-crbumgbamd0b6q6275pte0-alb1"
                load_balancer_hostname = ""
                name                   = ""
                resize                 = false
                state                  = "disabled"
            },
            {
                alb_type               = "private"
                disable_deployment     = false
                enable                 = false
                id                     = "private-crbumgbamd0b6q6275pte0-alb2"
                load_balancer_hostname = ""
                name                   = ""
                resize                 = false
                state                  = "disabled"
            },
            {
                alb_type               = "private"
                disable_deployment     = false
                enable                 = false
                id                     = "private-crbumgbamd0b6q6275pte0-alb3"
                load_balancer_hostname = ""
                name                   = ""
                resize                 = false
                state                  = "disabled"
            },
            {
                alb_type               = "public"
                disable_deployment     = false
                enable                 = true
                id                     = "public-crbumgbamd0b6q6275pte0-alb1"
                load_balancer_hostname = "18a7756c-us-south.lb.appdomain.cloud"
                name                   = ""
                resize                 = false
                state                  = "enabled"
            },
            {
                alb_type               = "public"
                disable_deployment     = false
                enable                 = true
                id                     = "public-crbumgbamd0b6q6275pte0-alb2"
                load_balancer_hostname = "18a7756c-us-south.lb.appdomain.cloud"
                name                   = ""
                resize                 = false
                state                  = "enabled"
            },
            {
                alb_type               = "public"
                disable_deployment     = false
                enable                 = true
                id                     = "public-crbumgbamd0b6q6275pte0-alb3"
                load_balancer_hostname = "18a7756c-us-south.lb.appdomain.cloud"
                name                   = ""
                resize                 = false
                state                  = "enabled"
            },
        ]
        crn                             = "crn:v1:bluemix:public:containers-kubernetes:us-south:a/6e532a4450dda5f0243d1a6c819f1197:bumgbamd0b6q6275pte0::"
        disable_public_service_endpoint = false
        flavor                          = "bx2.4x16"
        force_delete_storage            = false
        id                              = "bumgbamd0b6q6275pte0"
        ingress_hostname                = "icm-poc-vpc-cluster-3f3037ed650e84f558a8839c9ec8a6ed-0000.us-south.containers.appdomain.cloud"
        ingress_secret                  = (sensitive value)
      ~ kube_version                    = "1.18.10" -> "1.19"
        master_status                   = "Ready"
        master_url                      = "https://c113.us-south.containers.cloud.ibm.com:31576"
        name                            = "icm-poc-vpc-cluster"
        pod_subnet                      = "172.21.0.0/16"
        private_service_endpoint_url    = "https://c113.private.us-south.containers.cloud.ibm.com:31576"
        public_service_endpoint_url     = "https://c113.us-south.containers.cloud.ibm.com:31576"
        resource_controller_url         = "https://cloud.ibm.com/kubernetes/clusters"
        resource_crn                    = "crn:v1:bluemix:public:containers-kubernetes:us-south:a/6e532a4450dda5f0243d1a6c819f1197:bumgbamd0b6q6275pte0::"
        resource_group_id               = "af3e3dfffaf444579188b817bb7b320e"
        resource_group_name             = "Default"
        resource_name                   = "icm-poc-vpc-cluster"
        resource_status                 = "normal"
        service_subnet                  = "172.31.0.0/16"
        state                           = "normal"
        tags                            = []
        update_all_workers              = true
        vpc_id                          = "r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30"
        wait_for_worker_update          = true
        wait_till                       = "IngressReady"
        worker_count                    = 1
        worker_labels                   = {
            "ibm-cloud.kubernetes.io/worker-pool-id" = "bumgbamd0b6q6275pte0-f26133a"
            "type"                                   = "general"
        }

        zones {
            name      = "us-south-1"
            subnet_id = "0717-9623f8e0-82f6-440c-be17-593e9ec5f02e"
        }
        zones {
            name      = "us-south-2"
            subnet_id = "0727-54f8f9b8-6a11-448e-a4f4-7183f6ab6878"
        }
        zones {
            name      = "us-south-3"
            subnet_id = "0737-cf15cf67-02c4-4974-bb07-812a6ece71e0"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

ibm_container_vpc_cluster.cluster: Modifying... [id=bumgbamd0b6q6275pte0]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 1m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 1m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 1m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 1m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 1m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 1m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 2m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 2m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 2m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 2m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 2m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 2m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 3m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 3m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 3m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 3m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 3m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 3m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 4m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 4m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 4m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 4m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 4m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 4m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 5m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 5m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 5m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 5m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 5m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 5m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 6m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 6m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 6m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 6m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 6m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 6m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 7m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 7m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 7m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 7m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 7m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 7m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 8m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 8m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 8m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 8m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 8m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 8m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 9m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 9m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 9m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 9m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 9m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 9m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 10m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 10m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 10m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 10m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 10m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 10m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 11m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 11m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 11m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 11m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 11m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 11m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 12m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 12m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 12m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 12m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 12m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 12m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 13m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 13m10s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 13m20s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 13m30s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 13m40s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 13m50s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 14m0s elapsed]
ibm_container_vpc_cluster.cluster: Still modifying... [id=bumgbamd0b6q6275pte0, 14m10s elapsed]
ibm_container_vpc_cluster.cluster: Modifications complete after 14m16s [id=bumgbamd0b6q6275pte0]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Master version was updated, but no worker nodes were updated

$ ic ks cluster get --cluster icm-poc-vpc-cluster

Retrieving cluster icm-poc-vpc-cluster...
OK

Name:                           icm-poc-vpc-cluster
ID:                             bumgbamd0b6q6275pte0
State:                          normal
Status:                         All Workers Normal
Created:                        2020-11-12 11:51:38 +0200 (3 hours ago)
Resource Group ID:              af3e3dfffaf444579188b817bb7b320e
Resource Group Name:            Default
Pod Subnet:                     172.21.0.0/16
Service Subnet:                 172.31.0.0/16
Workers:                        6
Worker Zones:                   us-south-1, us-south-2, us-south-3
Ingress Subdomain:              icm-poc-vpc-cluster-3f3037ed650e84f558a8839c9ec8a6ed-0000.us-south.containers.appdomain.cloud
Ingress Secret:                 icm-poc-vpc-cluster-3f3037ed650e84f558a8839c9ec8a6ed-0000
Ingress Status:                 healthy
Ingress Message:                All Ingress components are healthy
Creator:                        -
Public Service Endpoint URL:    https://c113.us-south.containers.cloud.ibm.com:31576
Private Service Endpoint URL:   https://c113.private.us-south.containers.cloud.ibm.com:31576
Pull Secrets:                   enabled in the default namespace
VPCs:                           r006-3f0c5873-23ce-4aeb-a735-c1256f23fd30

Master
Status:     Ready (11 minutes ago)
State:      deployed
Health:     normal
Version:    1.19.3_1525
Location:   Dallas
URL:        https://c113.us-south.containers.cloud.ibm.com:31576

$ ic ks worker ls --cluster icm-poc-vpc-cluster

OK
ID                                                       Primary IP   Flavor     State    Status   Zone         Version
kube-bumgbamd0b6q6275pte0-icmpocvpccl-cassand-000007b3   10.10.0.10   cx2.2x4    normal   Ready    us-south-1   1.18.10_1532*
kube-bumgbamd0b6q6275pte0-icmpocvpccl-cassand-0000082c   10.30.0.10   cx2.2x4    normal   Ready    us-south-3   1.18.10_1532*
kube-bumgbamd0b6q6275pte0-icmpocvpccl-cassand-000009f9   10.20.0.9    cx2.2x4    normal   Ready    us-south-2   1.18.10_1532*
kube-bumgbamd0b6q6275pte0-icmpocvpccl-default-000001dd   10.20.0.7    bx2.4x16   normal   Ready    us-south-2   1.18.10_1532*
kube-bumgbamd0b6q6275pte0-icmpocvpccl-default-00000280   10.30.0.7    bx2.4x16   normal   Ready    us-south-3   1.18.10_1532*
kube-bumgbamd0b6q6275pte0-icmpocvpccl-default-00000344   10.10.0.7    bx2.4x16   normal   Ready    us-south-1   1.18.10_1532*

* To update to 1.19.3_1526 version, run 'ibmcloud ks worker update'. Review and make any required version changes before you update: 'https://ibm.biz/upworker'

[Anil-CM]

Hi @tpolekhin,

by default update_all_workers is set to false, i.e If there is a change in the kube_version it updates only the master node.
can you please check what value is set to update_all_workers?

[tpolekhin]

@Anil-CM we have both set to true

  update_all_workers     = true
  wait_for_worker_update = true

[tpolekhin]

@hkantare @Anil-CM I've tested new provider release 1.16.0 with this feature and can confirm that it's now working and upgrading worker nodes one by one.
However, on a pretty small cluster of 6 nodes it took a little bit over 3 hours to complete upgrade process.
This is a very long time, especially considering that most of the clusters are usually somewhere around 20-50 worker nodes.

Can we somehow speed up this process? Possibly upgrading multiple nodes at a time, but not all at once?
Maybe another approach could be used here?

VPC-Gen2 advertised a significant increase in workers provisioning as a part it's features, but according to my observations I can see that it still takes 20-30 minutes for a worker node to be replaced.

I would appreciate any suggestion you could give to speed up this process

[hkantare]

@tpolekhin its behind the scope of provider ..As per basic desgin of VPC Cluster the upgrade of nodes are in parallel and each nodes takes around 15 to 20 min.