Unsupported block type gke_backup_agent_config with gke-cluster and autopilot

[intotecho]

I am creating a gke_cluster with autopilot enabled:

module "my-gke-cluster" {
  source = "../cloud-foundation-fabric/modules/gke-cluster"
  project_id                = local.project_id
  name                      = local.cluster_name
  location                  = local.zone
  network                   = var.network_selflink
  subnetwork                = var.subnetwork_selflink
  secondary_range_pods      = "pods"
  secondary_range_services  = "services"
  default_max_pods_per_node = 32
  master_authorized_ranges = {
    internal-vms = "10.0.0.0/8"
  }
  private_cluster_config = {
    enable_private_nodes    = true
    enable_private_endpoint = true
    master_ipv4_cidr_block  = "192.168.0.0/28"
    master_global_access    = false
  }
  depends_on = [
    module.shared_vpc_access
  ]
  enable_autopilot  = true
  workload_identity = true
  release_channel   = "STABLE"
  addons = {
    cloudrun_config            = false
    dns_cache_config           = false
    horizontal_pod_autoscaling = true
    http_load_balancing        = true
    istio_config = {
      enabled = false
      tls     = false
    }
    network_policy_config                 = false
    gce_persistent_disk_csi_driver_config = false
    gcp_filestore_csi_driver_config       = false
    config_connector_config               = false
    kalm_config                           = false
    gke_backup_agent_config               = false
  }
}

But it gives error:

 terraform plan 
╷
│ Error: Unsupported block type
│
│   on ..\cloud-foundation-fabric\modules\gke-cluster\main.tf line 99, in resource "google_container_cluster" "cluster":
│   99:     gke_backup_agent_config {
│
│ Blocks of type "gke_backup_agent_config" are not expected here.

According to the google_container_cluster
the addons_config block supports:
...
gke_backup_agent_config - (Optional, Beta). The status of the Backup for GKE agent addon. It is disabled by default; Set enabled = true to enable.

I cannot pass an addon object without this flag.
"gke_backup_agent_config" is required.

I see that gke-cluster has provider=beta.

I am using the latest cloud-foundation-fabric branch
terraform version 1.2.3
providers:

├── provider[registry.terraform.io/hashicorp/google] ~> 4.17.0
├── provider[registry.terraform.io/hashicorp/google-beta] ~> 4.17.0
├── provider[terraform.io/builtin/terraform]
└── module.gis_project
    ├── provider[registry.terraform.io/hashicorp/random]
    ├── provider[registry.terraform.io/hashicorp/google]
    ├── provider[registry.terraform.io/hashicorp/google-beta]
    ├── module.gis_cluster
    │   ├── provider[registry.terraform.io/hashicorp/google]
    │   ├── module.geo-gke-cluster
    │   │   ├── provider[registry.terraform.io/hashicorp/google] >= 4.17.0
    │   │   └── provider[registry.terraform.io/hashicorp/google-beta] >= 4.17.0
    │   └── module.shared_vpc_access
    │       ├── provider[registry.terraform.io/hashicorp/google-beta] >= 3.43.0, < 5.0.0
    │       └── provider[registry.terraform.io/hashicorp/google] >= 3.43.0, < 5.0.0

My only workaround is to comment out this block in gke-cluster/main.tf:99

    gke_backup_agent_config {
      enabled = var.addons.gke_backup_agent_config
    }

Another workaround is to not enable autopilot.

[Edit]
Also, when I comment out the above statement, I get:

Error: Conflicting configuration arguments
│
│   with module.gis_project.module.gis_cluster.module.geo-gke-cluster.google_container_cluster.cluster,
│   on ..\..\00_-_modules\cloud-foundation-fabric\modules\gke-cluster\main.tf line 90, in resource "google_container_cluster" "cluster":
│   90:     gcp_filestore_csi_driver_config {
│
│ "addons_config.0.gcp_filestore_csi_driver_config": conflicts with enable_autopilot
╵

Removing the entire addons block in the call leads to:

│ Error: Conflicting configuration arguments
│
│   with module.gis_project.module.gis_cluster.module.geo-gke-cluster.google_container_cluster.cluster,
│   on ..\..\00_-_modules\cloud-foundation-fabric\modules\gke-cluster\main.tf line 90, in resource "google_container_cluster" "cluster":
│   90:     gcp_filestore_csi_driver_config {
│
│ "addons_config.0.gcp_filestore_csi_driver_config": conflicts with enable_autopilot

[juliocc]

Seems like the gke_backup_agent_config block was added in version 4.19 of the beta provider. We'll upload a fix shortly

As a quick workaround you can use terraform init -upgrade=true to tell terraform to update all providers to their latest version

[intotecho]

Thanks, I have pulled #706
Updated required google providers to 4.20.0 and run terraform init -upgrade=true
│ Error: Conflicting configuration arguments
│
│ with module.gis_project.module.gis_cluster.module.geo-gke-cluster.google_container_cluster.cluster,
│ on ....\00_-_modules\cloud-foundation-fabric\modules\gke-cluster\main.tf line 90, in resource "google_container_cluster" "cluster":
│ 90: gcp_filestore_csi_driver_config {
│
│ "addons_config.0.gcp_filestore_csi_driver_config": conflicts with enable_autopilot
whether I specify true, or false or leave out addons.

[juliocc]

@intotecho the issue should be fixed in the master branch now. Thanks for the report!

[juliocc]

Thanks, I have pulled #706 Updated required
(removed)
│ "addons_config.0.gcp_filestore_csi_driver_config": conflicts with enable_autopilot
whether I specify true, or false or leave out addons.

So this is a different problem. Let me take alook

[juliocc]

@intotecho I completely missed the second part of your original report. I think it should be fixed now, can you give it another try?

[intotecho]

I have pulled #707 with the same result. have the changes been pushed?
I used the same addons as in the test fixture

│ Error: googleapi: Error 400: Addons {"gce-persistent-disk-csi-driver"} are required to be enabled for Autopilot clusters., badRequest
│
│   with module.gis_project.module.gis_cluster.module.geo-gke-cluster.google_container_cluster.cluster,
│   on ..\..\00_-_modules\cloud-foundation-fabric\modules\gke-cluster\main.tf line 32, in resource "google_container_cluster" "cluster":
│   32: resource "google_container_cluster" "cluster" {

The plan includes

 + addons_config {
          + cloudrun_config {
              + disabled = true
            }

          + config_connector_config {
              + enabled = false
            }

          + dns_cache_config {
              + enabled = (known after apply)
            }

          + gce_persistent_disk_csi_driver_config {
              + enabled = false
            }

          + gcp_filestore_csi_driver_config {
              + enabled = (known after apply)
            }

          + gke_backup_agent_config {
              + enabled = false
            }

          + horizontal_pod_autoscaling {
              + disabled = false
            }

          + http_load_balancing {
              + disabled = false
            }

          + istio_config {
              + disabled = true
            }

          + kalm_config {
              + enabled = false
            }

          + network_policy_config {
              + disabled = (known after apply)
            }
        }

I see that the test module/gke-cluster passes, so not sure why I am stuck.

[juliocc]

can you try with gce_persistent_disk_csi_driver_config = true?

[intotecho]

No change. Even this fails if I modify gke-cluster/main.tf to force enabled=true

dynamic "gcp_filestore_csi_driver_config" {
      # Pass the user-provided value when autopilot is disabled. When
      # autopilot is enabled, pass the value only when the addon is
      # set to true. This will fail but warns the user that autopilot
      # doesn't support this option, instead of silently discarding
      # and hiding the error
      for_each = !var.enable_autopilot || (var.enable_autopilot && var.addons.gcp_filestore_csi_driver_config) ? [""] : []
      content {
        enabled = true **#var.addons.gcp_filestore_csi_driver_config**
      }
    }

Same if I hard code it to false.
What does work is to comment out the whole block.

/* dynamic "gcp_filestore_csi_driver_config" {
      # Pass the user-provided value when autopilot is disabled. When
      # autopilot is enabled, pass the value only when the addon is
      # set to true. This will fail but warns the user that autopilot
      # doesn't support this option, instead of silently discarding
      # and hiding the error
      for_each = !var.enable_autopilot || (var.enable_autopilot && var.addons.gcp_filestore_csi_driver_config) ? [""] : []
      content {
        enabled = var.addons.gcp_filestore_csi_driver_config
      }
    } */

[juliocc]

@intotecho we tried a different approach with PR #708. Can you try once again?

Thanks, and sorry for the back and forth

[intotecho]

Thanks for the PR. Same error sorry.

Error: Conflicting configuration arguments
│
│   with module.gis_project.module.gis_cluster.module.geo-gke-cluster.google_container_cluster.cluster,
│   on ..\..\00_-_modules\cloud-foundation-fabric\modules\gke-cluster\main.tf line 61, in resource "google_container_cluster" "cluster":
│   61:   addons_config {
│
│ "addons_config.0.gcp_filestore_csi_driver_config": conflicts with enable_autopilot

However, this works. edit to force each =[]

dynamic "gcp_filestore_csi_driver_config" {
      # Pass the user-provided value when autopilot is disabled. When
      # autopilot is enabled, pass the value only when the addon is
      # set to true. This will fail but warns the user that autopilot
      # doesn't support this option, instead of silently discarding
      # and hiding the error
      #for_each = var.enable_autopilot && !var.addons.gcp_filestore_csi_driver_config ? [] : [""]
      for_each = []
      content {
        enabled = var.addons.gcp_filestore_csi_driver_config
      }
    }

And DeMorgan's theorem also works (changing && to ||)

dynamic "gcp_filestore_csi_driver_config" {
# Pass the user-provided value when autopilot is disabled. When
# autopilot is enabled, pass the value only when the addon is
# set to true. This will fail but warns the user that autopilot
# doesn't support this option, instead of silently discarding
# and hiding the error
for_each = var.enable_autopilot || !var.addons.gcp_filestore_csi_driver_config ? [] : [""]
#for_each = []
content {
enabled = var.addons.gcp_filestore_csi_driver_config
}
}

[juliocc]

We tried the code you provided in your initial comment and it worked. Are you still doing the same thing?

[intotecho]

The PR is working fine as is.
I had set the addons.gcp_filestore_csi_driver_config = true, and that triggered the error - as expected!
Thanks for the updates. It can be closed.