ERROR Reconciler error {"controller": "cert-rotator", "object": {"name":"kedaorg-certs","namespace":"keda"}

[ori-21]

Report

Hi all,
i am facing an issue that i hope anybody here have experienced and would help me.
I upgrated keda on my aks cluster from 2.8.2 to 2.13.0 version and from keda-operator pod i get logs as below:

Expected Behavior

ValidatedWebhookConfiguration

Actual Behavior

there is an error updating webhook with certificate

Steps to Reproduce the Problem

1.upgrade from keda 2.8.1 to keda 2.13.0

Logs from KEDA operator

/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

2024-02-28T02:03:16Z ERROR Reconciler error {"controller": "cert-rotator", "object": {"name":"kedaorg-certs","namespace":"keda"}, "namespace": "keda", "name": "kedaorg-certs", "reconcileID": "5ef2b440-11bd-489e-a384-d9f3768fbc95", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-02-28T02:03:16Z INFO cert-rotation no cert refresh needed
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-02-28T02:03:16Z INFO cert-rotation no cert refresh needed
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-02-28T02:03:16Z INFO cert-rotation no cert refresh needed
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-02-28T02:03:16Z INFO cert-rotation no cert refresh needed
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-02-28T02:03:16Z INFO cert-rotation no cert refresh needed
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-02-28T02:03:16Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-02-28T02:03:16Z ERROR cert-rotation Error updating webhook with certificate {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:839
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:785
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-02-28T02:03:16Z ERROR Reconciler error {"controller": "cert-rotator", "object": {"name":"kedaorg-certs","namespace":"keda"}, "namespace": "keda", "name": "kedaorg-certs", "reconcileID": "dfabd7a8-40ef-4154-b651-c6aa6b9dd0ee", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

KEDA Version

2.13.0

Kubernetes Version

1.27

Platform

Microsoft Azure

Scaler Details

No response

Anything else?

No response

[ori-21]

KEDA is deployed with helm from this repo https://kedacore.github.io/charts

[kbocock-krg]

More info we are also using the same chart.
"Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

[regisferlima]

I'm facing the same issue... :(

[forzamehlano]

+1

[vinayak-shanawad]

+1

[vinayak-shanawad]

I see this issue does not persist in Keda 2.14.0 and chart version: 2.14.2

[zroubalik]

@vinayak-shanawad thanks for the confirmation.

[vinayak-shanawad]

@zroubalik It works fine in my local kind cluster but not in our AWS EKS cluster because we already using Datadog as external metrics server, we hit this issue now.

[sohel2020]

@vinayak-shanawad thanks for the confirmation.

It does. I'm using 2.14.2 in k8s v1.28.11

[sohel2020]

2024-06-26T19:08:54Z ERROR cert-rotation Error updating webhook with certificate {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "error": "Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again"}

[vinayak-shanawad]

@sohel2020 Are you getting this error from a local kind cluster?

[virasana]

I see this issue does not persist in Keda 2.14.0 and chart version: 2.14.2

Unfortunately, the issue is still happening for me on keda 2.14.0 and chart version 2.14.2
We are using AKS v1.28.9

Did you destroy your helm release or did you upgrade in place?

[jfouche-vendavo]

Confirmed that this issue still occurs for me on a number of clusters in AKS.
Helm chart version: keda-2.15.0
App Version: 2.15.0
I destroyed the helm deployment and reinstalled from scratch.
@zroubalik Any ideas? This appears to be an issue experienced by others too?
Many Thanks

[jfouche-vendavo]

Does anybody have insight as to why the error response from the API?
It appears that this can occur if yaml with inappropriate field metadata (such as resourceVersion, timestamp) are applied in a yaml file? See for example: https://stackoverflow.com/questions/51297136/kubectl-error-the-object-has-been-modified-please-apply-your-changes-to-the-la

I am using AKS - is it possible that AKS could be interfering with the certs rotation on the APIService object e.g. by attempting to update the caBundle? See Azure Docs here:

[JorTurFer]

This error is transitory until KEDA operator is able to configure the required services. If you see it during a few minutes, it's totally normal. if you see that the error persists, maybe there is any other reconcile (such as ArgoCD or Flux) modifying the manifests and being in conflict with KEDA (because KEDA patches the manifest to include the caBundle and those tools can try to remove it).

If you are using ArgoCD with autosync or flux, I'd suggest including a rule to skip the caBundle from APIService and ValidatingWebhookConfiguration. You can also use cert-manager to generate and patch the required resources instead of using KEDA internal service (cert-manager is better although KEDA brings the basic setup) -> https://keda.sh/blog/2023-05-02-certificate-improvements/

[jfouche-vendavo]

Thank you @JorTurFer . Yes, the error is transitory. However, it is also a breaking error (see KubeAggregatedAPIErrors here). We see interruption to the Kubernetes Aggregated API (FailedDiscoveryCheck), which means that Prometheus metrics are not propagated for the duration. In other words, it is affecting the aggregated API on the cluster in general.

As above, possibly the error is because Azure AKS is trying to manage the caBundle, thus clashing with keda-operator? (could this be possible?)

Thank you for the above advice. We will disable the Keda certs rotation - I believe that this can be done by setting certificates.autoGenerated=false in the helm chart values. This, along with deployment of cert-manager will possibly be a fix.

May I suggest that this behaviour is a bug in Keda which deserves some attention?

Many Thanks

[jfouche-vendavo]

UPDATE: FYI
@JorTurFer
FYI I have disabled cert rotation as above but this does not fix the KubeAggregatedAPIErrors. These errors must be happening elsewhere.
Thanks for your help.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[pandvag]

please reopen, since this error persists even in 2.16.0

[JorTurFer]

please reopen, since this error persists even in 2.16.0

This was closed by inactivity. Could you open a new issue to track your case?