Status info missing from kubernetes_manifest object attribute after apply

[davidalger]

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.1.4
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v4.10.0
+ provider registry.terraform.io/hashicorp/helm v2.4.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.8.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

Kubernetes version 1.21

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "prometheus_server_serviceattachment" {
  count = var.psc_enabled ? 1 : 0

  manifest = {
    apiVersion = "networking.gke.io/v1beta1"
    kind       = "ServiceAttachment"

    metadata = {
      name      = "${var.release_name}-server"
      namespace = kubernetes_namespace.this.id
    }

    spec = {
      connectionPreference = "ACCEPT_AUTOMATIC"
      natSubnets = [
        google_compute_subnetwork.prometheus_server_psc_subnet[0].name
      ]
      proxyProtocol = false
      resourceRef = {
        kind = "Service"
        name = kubernetes_service.prometheus_server_lb[0].metadata[0].name
      }
    }
  }

  wait_for = {
    fields = {
      "status.serviceAttachmentURL" = "^https:\\/\\/www.googleapis.com\\/.*\\/serviceAttachments\\/.*$"
    }
  }
}

Debug Output

$ tg console
> kubernetes_manifest.prometheus_server_serviceattachment[0].object
{
  "apiVersion" = "networking.gke.io/v1beta1"
  "kind" = "ServiceAttachment"
  "metadata" = {
    "annotations" = tomap(null) /* of string */
    "clusterName" = tostring(null)
    "creationTimestamp" = tostring(null)
    "deletionGracePeriodSeconds" = tonumber(null)
    "deletionTimestamp" = tostring(null)
    "finalizers" = tolist([
      "networking.gke.io/service-attachment-finalizer",
    ])
    "generateName" = tostring(null)
    "generation" = tonumber(null)
    "labels" = tomap(null) /* of string */
    "managedFields" = null /* tuple */
    "name" = "prometheus-server"
    "namespace" = "prometheus"
    "ownerReferences" = tolist(null) /* of object */
    "resourceVersion" = tostring(null)
    "selfLink" = tostring(null)
    "uid" = tostring(null)
  }
  "spec" = {
    "connectionPreference" = "ACCEPT_AUTOMATIC"
    "consumerAllowList" = tolist(null) /* of object */
    "consumerRejectList" = tolist(null) /* of string */
    "natSubnets" = tolist([
      "prometheus-server-psc-subnet-5c4bdd",
    ])
    "proxyProtocol" = false
    "resourceRef" = {
      "apiGroup" = tostring(null)
      "kind" = "Service"
      "name" = "prometheus-server-lb"
    }
  }
}

Expected Behavior

The documentation states "the object attribute contains the state of the resource as returned by the Kubernetes API, including all default values."

Various objects will expose valuable information in their status. In this case, I'd like to be able to have the Terraform module export the service attachment URL as an output:

output "service_attachment_url" {
  value = kubernetes_manifest.prometheus_server_serviceattachment.object.status.serviceAttachmentURL
}

The output should contain the same value that is shown here when getting the object via kubectl:

$ kubectl -n prometheus get serviceattachment prometheus-server -o yaml
apiVersion: networking.gke.io/v1beta1
kind: ServiceAttachment
metadata:
  creationTimestamp: "2022-02-15T17:55:11Z"
  finalizers:
  - networking.gke.io/service-attachment-finalizer
  generation: 3
  name: prometheus-server
  namespace: prometheus
  resourceVersion: "8019482"
  uid: 64e16c47-26b5-4dd3-aff7-289fed47bc29
spec:
  connectionPreference: ACCEPT_AUTOMATIC
  natSubnets:
  - prometheus-server-psc-subnet-5c4bdd
  proxyProtocol: false
  resourceRef:
    kind: Service
    name: prometheus-server-lb
status:
  forwardingRuleURL: https://www.googleapis.com/compute/beta/projects/my-project/regions/us-west1/forwardingRules/ad3bccdeca9114142acdc91ddcb5b702
  lastModifiedTimestamp: "2022-02-15T17:55:15Z"
  serviceAttachmentURL: https://www.googleapis.com/compute/beta/projects/my-project/regions/us-west1/serviceAttachments/k8s1-sa-hl4gna7v-prometheus-prometheus-server-as5d6ghe

Actual Behavior

The resources is created, but there is no way to access the serviceAttachmentURL value in Terraform.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[alexsomesan]

Hi,

The "status" attribute is removed by the provider logic on purpose. This is becuase the value of status can change on the API side without intervention from Terraform and thus produce unactionable diffs in the following plan operations.
The status attribute is better reflected through a data-source (read only) and this will be available once PR #1548 will be released.

Let me know if this doesn't sufficiently clarify the issue.

[davidalger]

the value of status can change on the API side without intervention from Terraform

Understand this, both status and annotations frequently change after initial resource creation, especially for objects which have custom controllers (Ingress and ServiceAttachement being two examples) that create resources within a given cloud provider's system to provide X functionality.

thus produce unactionable diffs in the following plan operations
The status attribute is better reflected through a data-source (read only)

This is where I would beg to differ. The diff would be one of a state change, it should show up as a "change made outside of Terraform" and when tf apply or tf apply -refresh-only is run the diff should come in and be reflected in state.

One should not need to have a data resource that depends on a resource you just created to get a value from the created resource.

An existing example where this works as one would naturally have expected here is the kubernetes_ingress_v1 resource. You can specify wait_for_load_balancer = true and it's creation will block until status[0].load_balancer is no longer empty, allowing you to use the value the ingress controller added to the status object.

If you omit wait_for_load_balancer so the resource creation doesn't wait, indicating completion as soon as the object is applied to k8s, the following happens:

a) Initial object is applied and the Ingress returned from the API looks like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: gce
    networking.gke.io/v1beta1.FrontendConfig: traefik
  creationTimestamp: "2022-02-17T18:10:09Z"
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 1
  name: traefik
  namespace: traefik
  resourceVersion: "612524"
  uid: 85d6bb25-8ed3-4389-8fc1-2d4c97df8e2c
spec:
  defaultBackend:
    service:
      name: traefik
      port:
        name: web
  tls:
  - secretName: traefik-tls-algerdev-ptlabs-dev
status:
  loadBalancer: {}

b) After some short amount of time — in this case, once the controller has created an http/s load balancer on Google Cloud — the object is updated by the controller and then looks like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/backends: '{"k8s1-9fd681d1-traefik-traefik-80-01cfbd6f":"Unknown"}'
    ingress.kubernetes.io/forwarding-rule: k8s2-fr-hs6u9rbo-traefik-traefik-35kfb15b
    ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-hs6u9rbo-traefik-traefik-35kfb15b
    ingress.kubernetes.io/https-target-proxy: k8s2-ts-hs6u9rbo-traefik-traefik-35kfb15b
    ingress.kubernetes.io/redirect-url-map: k8s2-rm-hs6u9rbo-traefik-traefik-35kfb15b
    ingress.kubernetes.io/ssl-cert: k8s2-cr-hs6u9rbo-kwnt3rdrvy2od6sz-66091e840b26166a
    ingress.kubernetes.io/static-ip: k8s2-fr-hs6u9rbo-traefik-traefik-35kfb15b
    ingress.kubernetes.io/target-proxy: k8s2-tp-hs6u9rbo-traefik-traefik-35kfb15b
    ingress.kubernetes.io/url-map: k8s2-um-hs6u9rbo-traefik-traefik-35kfb15b
    kubernetes.io/ingress.class: gce
    networking.gke.io/v1beta1.FrontendConfig: traefik
  creationTimestamp: "2022-02-17T18:10:09Z"
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 1
  name: traefik
  namespace: traefik
  resourceVersion: "613099"
  uid: 85d6bb25-8ed3-4389-8fc1-2d4c97df8e2c
spec:
  defaultBackend:
    service:
      name: traefik
      port:
        name: web
  tls:
  - secretName: traefik-tls-algerdev-ptlabs-dev
status:
  loadBalancer:
    ingress:
    - ip: 34.149.125.208

c) Run tf apply and you will see the change to the object's status will show up as a change made outside of Terraform:

...
kubernetes_ingress_v1.traefik: Refreshing state... [id=traefik/traefik]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # kubernetes_ingress_v1.traefik has changed
  ~ resource "kubernetes_ingress_v1" "traefik" {
        id     = "traefik/traefik"
      ~ status = [
          ~ {
              ~ load_balancer = [
                  ~ {
                      ~ ingress = [
                          + {
                              + hostname = ""
                              + ip       = "35.241.49.191"
                            },
                        ]
                    },
                ]
            },
        ]

      ~ metadata {
            name             = "traefik"
          ~ resource_version = "646816" -> "647519"
            # (5 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like
to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only
Releasing state lock. This may take a few moments...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

The new value the controller added to status is now stored in the Terraform state, even though the resource will never change it, and it's available to use via remote state, etc.

In the case of kubernetes_manifest the wait_for field exists, as can be seen in the example I originally posted, to block until this value is set by the controller which manages these objects on the cluster:

  wait_for = {
    fields = {
      "status.serviceAttachmentURL" = "^https:\\/\\/www.googleapis.com\\/.*\\/serviceAttachments\\/.*$"
    }
  }

The examples in the documentation show configuration which basically does the same as wait_for_load_balancer on the ingress resource:

      # Check an ingress has an IP
      "status.loadBalancer.ingress[0].ip" = "^(\\d+(\\.|$)){4}"

It stands to reason that if one is to block create and update calls until a field is set or has a particular value, that the value will be available on the object when the call completes, and not require a data resource to fetch it. The data resource is valuable as well — don't get me wrong here — but it's use-case is slightly different in that it can be used to fetch information about something created outside the confines of the current Terraform module. It shouldn't be needed to gather info on something just created in the same module.

[shankerwangmiao]

Hi,

The "status" attribute is removed by the provider logic on purpose. This is becuase the value of status can change on the API side without intervention from Terraform and thus produce unactionable diffs in the following plan operations. The status attribute is better reflected through a data-source (read only) and this will be available once PR #1548 will be released.

Let me know if this doesn't sufficiently clarify the issue.

Hi, however in PR #1548, the status attribute is also deleted via RemoveServerSideFields and unavailable via the data-source.

[pisto]

see #1699

[github-actions]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[davidalger]

This really needs to be addressed, not left to be closed.

[iBrandyJackson]

We believe this issue has been resolved with PR #1548 and PR #1699. If this problem persists, please reopen a new issue and link back to this one. Thanks!