couldn't get resource list for webhook.certmanager.k8s.io/v1beta1

[aungzy]

Describe the bug
After installing flux in OpenShift 4.2 cluster, the operator was able to refresh git repository, but not able to sync resources.
I am seeing the following 2 error messages:

type="internal kubernetes error"
kubernetes_caller=k8s.io/[email protected]+incompatible/discovery/cached/memory/memcache.go:111
err="couldn't get resource list for webhook.certmanager.k8s.io/v1beta1: the server is currently unable to handle the request"

component=sync-loop
err="collating resources in cluster for sync: unable to retrieve the complete list of server APIs: webhook.certmanager.k8s.io/v1beta1: the server is currently unable to handle the request"

To Reproduce
Steps to reproduce the behaviour:
0. What's your setup? As per https://docs.fluxcd.io/en/latest/tutorials/get-started.html

Expected behavior
Flux should be able to successfully run kubectl apply command to sync resources.

Additional context

• Flux version: 1.15.0
• Kubernetes version: 1.14 (with OpenShift 4.2)
• Git provider: Gitlab CE

[martintomac]

I had the same issue. Upgrading cert-manager solved the issue for me.

Additional context

Flux version: 1.15.0
Kubernetes version: 1.16 (Rancher)
Git provider: Gitlab CE

[ambis]

Same problem here. Started out of the blue as per the logs (2019-11-15 midnight). No changes in flux source repo, none of the cluster components other than my app itself is automated for updates.

Running k3s v0.10.2, cert manager 0.11.0 and flux 1.15.0.

I deleted cert-manager completely with delete -f and flux reapplied it (now that it started working), and boom, here comes the error again.

[cmanzi]

Can confirm I'm seeing this with Flux 1.17.0 as well. Both cert-manager and metrics-server API services. Restarting the Flux pod seems to get fix it.

[2opremio]

I will make sure we exclude webhook.certmanager.k8s.io resources by default, and make the excluded resources configurable as per #2642

[2opremio]

Note that this is a problem with Kubernetes or the configuration of Kubernetes itself, which fails to answer List requests for webhook.certmanager.k8s.io/v1beta1

[aungzy]

Thanks @2opremio !

[clearclaw]

ETA on this?

[2opremio]

I am going to work on this tomorrow, can't promise a specific date when it will be released.

[clearclaw]

Thanks. We use cert-manager heavily and this is rather balling things up while I'm trying to sell the team on Flux, so a quick point release would be appreciated.

[stefanprodan]

@clearclaw having Flux ignore the webhook resource doesn't solve the underlying issue with your cluster. My guess is that either your cluster etcd is corrupted or you need to create a firewall rule see cert-manager/cert-manager#2109

[clearclaw]

Ahh, excellent pointer! And yeah, we're using private GKE clustes with private nodes, which breaks a few common assumptions about network accessibility. Good catch. Thanks.