Kong missing ClusterRole permission

[meeech]

What Happened?

minikube version: v1.33.1
commit: 5883c09

Turn on Kong Plugin. Setup ingress for a deployment. Kong pod kept crashing.

Looks like missing permission. Was able to fix by applying the missing permissions

- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["list"]

to the kong-serviceaccount and then things worked as expected.
(pr incoming)

Attach the log file

time="2024-06-01T19:13:44Z" level=info msg="Starting Controller" logger=controllers.KongAdminAPIService
time="2024-06-01T19:13:44Z" level=info msg="Starting EventSource" logger=controllers.KongClusterPlugin source="kind source: *v1.IngressClass"
time="2024-06-01T19:13:44Z" level=info msg="Starting EventSource" logger=controllers.KongClusterPlugin source="kind source: *v1.KongClusterPlugin"
time="2024-06-01T19:13:44Z" level=info msg="Starting Controller" logger=controllers.KongClusterPlugin
W0601 19:13:44.811593       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:13:44.811626       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
time="2024-06-01T19:13:44Z" level=error msg="failed to list classless kongconsumergroups" error="no matches for kind \"KongConsumerGroup\" in version \"configuration.konghq.com/v1beta1\"" logger=controllers.KongConsumerGroup
W0601 19:13:45.914984       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:13:45.915034       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
time="2024-06-01T19:13:47Z" level=info msg="successfully synced configuration to Kong" update_strategy=InMemory url="https://10.244.3.172:8444"
W0601 19:13:47.909726       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:13:47.909769       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
W0601 19:13:52.366098       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:13:52.366128       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
W0601 19:14:01.086791       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:14:01.086857       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
    >  goroutine 339 [running]:
    >  runtime/debug.Stack()
    >      /usr/local/go/src/runtime/debug/stack.go:24 +0x64
    >  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
    >      /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/log.go:60 +0xf4
    >  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error(0x4000a96d00, {0x1c328c0, 0x4001065100}, {0x19b5f1d, 0x3d}, {0x4000fa61e0, 0x2, 0x2})
    >      /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/deleg.go:139 +0x44
    >  github.com/go-logr/logr.Logger.Error({{0x1c50d28?, 0x4000a96d00?}, 0x1556580?}, {0x1c328c0, 0x4001065100}, {0x19b5f1d, 0x3d}, {0x4000fa61e0, 0x2, 0x2})
    >      /go/pkg/mod/github.com/go-logr/[email protected]/logr.go:299 +0xb4
    >  sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1({0x1c4d3f0?, 0x40005a76d0?})
    >      /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:63 +0x25c
    >  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2(0x4000bb5dc0?, {0x1c4d3f0?, 0x40005a76d0?})
    >      /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:73 +0x58
    >  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x1c4d3f0, 0x40005a76d0}, {0x1c43d50?, 0x400059e860}, 0x1, 0x0, 0x0?)
    >      /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/loop.go:74 +0x1ac
    >  k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel({0x1c4d3f0, 0x40005a76d0}, 0x0?, 0x0?, 0x400089a6f8?)
    >      /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/poll.go:33 +0x74
    >  sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1()
    >      /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:56 +0xb8
    >  created by sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start in goroutine 299
    >      /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:48 +0x1c4
W0601 19:14:22.127772       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:14:22.127911       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
W0601 19:14:49.813404       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:14:49.813508       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
W0601 19:15:30.788765       1 reflector.go:535] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E0601 19:15:30.789046       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kong:kong-serviceaccount" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
time="2024-06-01T19:15:44Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for KongV1KongClusterPlugin caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass" logger=controllers.KongClusterPlugin
time="2024-06-01T19:15:44Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for KongV1Beta1KongConsumerGroup caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass" logger=controllers.KongConsumerGroup
time="2024-06-01T19:15:44Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for CoreV1Secret caches to sync: timed out waiting for cache to be synced for Kind *v1.Secret" logger=controllers.Secrets
time="2024-06-01T19:15:44Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for KongV1KongConsumer caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass" logger=controllers.KongConsumer
time="2024-06-01T19:15:44Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for DynamicCRDController caches to sync: timed out waiting for cache to be synced for Kind *v1.CustomResourceDefinition" logger=controllers.Dynamic/HTTPRoute
time="2024-06-01T19:15:44Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for KongAdminAPIEndpoints caches to sync: timed out waiting for cache to be synced for Kind *v1.EndpointSlice" logger=controllers.KongAdminAPIService
time="2024-06-01T19:15:44Z" level=info msg="Stopping and waiting for non leader election runnables"
time="2024-06-01T19:15:44Z" level=info msg="Stopping and waiting for leader election runnables"
time="2024-06-01T19:15:44Z" level=error msg="error received after stop sequence was engaged" error="failed to wait for KongV1Beta1KongConsumerGroup caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"
time="2024-06-01T19:15:44Z" level=error msg="error received after stop sequence was engaged" error="failed to wait for CoreV1Secret caches to sync: timed out waiting for cache to be synced for Kind *v1.Secret"
time="2024-06-01T19:15:44Z" level=error msg="error received after stop sequence was engaged" error="failed to wait for KongV1KongConsumer caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"
time="2024-06-01T19:15:44Z" level=error msg="error received after stop sequence was engaged" error="failed to wait for DynamicCRDController caches to sync: timed out waiting for cache to be synced for Kind *v1.CustomResourceDefinition"
time="2024-06-01T19:15:44Z" level=error msg="error received after stop sequence was engaged" error="failed to wait for KongAdminAPIEndpoints caches to sync: timed out waiting for cache to be synced for Kind *v1.EndpointSlice"
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.UDPIngress worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.UDPIngress
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.UDPIngress
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.EndpointSlice worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.EndpointSlice
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.Ingress.netv1 worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.Ingress.netv1
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.Ingress.netv1
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.KongPlugin worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.KongPlugin
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.KongPlugin
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.TCPIngress worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.TCPIngress
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.TCPIngress
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.Service worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.Service
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.KongIngress worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.KongIngress
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.KongIngress
time="2024-06-01T19:15:44Z" level=error msg="could not update kong admin" error="performing update for https://10.244.3.172:8444 failed: failed to verify kong readiness: making HTTP request: Get \"https://10.244.3.172:8444/status\": context canceled\nperforming update for https://10.244.3.172:8444 failed: failed to verify kong readiness: making HTTP request: Get \"https://10.244.3.172:8444/status\": context canceled" subsystem=dataplane-synchronizer
time="2024-06-01T19:15:44Z" level=info msg="context done: shutting down the proxy update server" subsystem=dataplane-synchronizer
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.Dynamic/Gateway worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.Dynamic/Gateway
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.Dynamic/Gateway
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.Dynamic/KnativeV1Alpha1/Ingress worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.IngressClass.netv1 worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.Dynamic/KnativeV1Alpha1/Ingress
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.IngressClass.netv1
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.Dynamic/KnativeV1Alpha1/Ingress
time="2024-06-01T19:15:44Z" level=info msg="Starting workers" logger=controllers.IngressClassParameters worker count=1
time="2024-06-01T19:15:44Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.IngressClassParameters
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.IngressClassParameters
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.Service
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.IngressClass.netv1
time="2024-06-01T19:15:44Z" level=info msg="All workers finished" logger=controllers.EndpointSlice
time="2024-06-01T19:15:44Z" level=info msg="Stopping and waiting for caches"
time="2024-06-01T19:15:44Z" level=info msg="Stopping and waiting for webhooks"
time="2024-06-01T19:15:44Z" level=info msg="Stopping and waiting for HTTP servers"
time="2024-06-01T19:15:44Z" level=info msg="Wait completed, proceeding to shutdown the manager"
time="2024-06-01T19:15:44Z" level=info msg="stopping telemetry manager"
Error: failed to wait for KongV1KongClusterPlugin caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass
Stream closed EOF for kong/ingress-kong-667888bfcf-9jk29 (ingress-controller)

Operating System

macOS (Default)

Driver

Docker