autoscaling.keda.sh/paused-replicas Automatic scaling was not restored after test deletion in 1.8.2

[LY-today]

Report

autoscaling.keda.sh/paused-replicas Automatic scaling was not restored after test deletion in 1.8.2

Expected Behavior

autoscaling.keda.sh/paused-replicas Automatic scaling was restored after test deletion in 1.8.2

Actual Behavior

autoscaling.keda.sh/paused-replicas Automatic scaling was not restored after test deletion in 1.8.2

Steps to Reproduce the Problem

1.kubectl apply --server-side -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml
2.add annotations autoscaling.keda.sh/paused-replicas: '11'
3.delete annotations autoscaling.keda.sh/paused-replicas: '11'
4.hpa maxReplicaCount and minReplicaCount not change

Logs from KEDA operator

example

KEDA Version

< 2.11.0

Kubernetes Version

Other

Platform

None

Scaler Details

cron

Anything else?

$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.21.1
WARNING: version difference between client (1.29) and server (1.21) exceeds the supported minor version skew of +/-1

kubectl apply --server-side -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml

[tomkerkhove]

KEDA v2.8 is no longer supported; can you please try with KEDA 2.15?

[LY-today]

KEDA v2.8 is no longer supported; can you please try with KEDA 2.15?

Unfortunately, our internal cluster version is v1.21.1. According to the official recommendation of keda https://keda.sh/docs/2.15/operate/cluster/#kubernetes-compatibility, we use keda’s v2.8.2. I also tested it. After version 2.15 of keda, the program will panic because there is no v2 version of hpa, which can be understood as being incompatible with 1.21.1

[LY-today]

@tomkerkhove https://keda.sh/docs/2.8/concepts/scaling-deployments/#pause-autoscaling，Didn’t the official support paused-replicas in v2.8.2? So is this a bug?

[tomkerkhove]

KEDA 2.8.2 is from Jan 2023; I doubt that we will issue a patch for it: https://github.com/kedacore/keda/releases/tag/v2.8.2

[LY-today]

@tomkerkhove So in the 2.8.2 version I am currently testing, adding paused-replicas is in line with expectations, but deleting paused-replicas is not in line with expectations. Is it really a bug?

[LY-today]

@tomkerkhove So in the 2.8.2 version I am currently testing, adding paused-replicas is in line with expectations, but deleting paused-replicas is not in line with expectations. Is it really a bug?

@tomkerkhove The purpose of trying to determine whether this problem is a bug is to clarify whether it is caused by my non-compliant operation or whether it is a defect of 2.8.2, because I see that the official clearly shows this function to users.

[JorTurFer]

I remember some issues related with the pause feature in the beginning due to errors on (internal) selectors. To test if this is your case, restart the KEDA operator pod, if it fixes the HPA, you are affected by these problems

[LY-today]

I remember some issues related with the pause feature in the beginning due to errors on (internal) selectors. To test if this is your case, restart the KEDA operator pod, if it fixes the HPA, you are affected by these problems

@JorTurFer Restart keda-metrics-apiserver or keda-operator pod?

[JorTurFer]

keda-operator pod

[LY-today]

keda-operator pod

After deleting autoscaling.keda.sh/paused-replicas, delete the pod of keda-operator and restart it, hpa resumes Automatic scaling.But this shouldn't be a routine process, is there any other solution? I also saw the design of autoscaling.keda.sh/paused, can it solve my problem?

[JorTurFer]

But this shouldn't be a routine process, is there any other solution?

This bug is already solved in latest versions. I fully understand that you are attached to v2.8 as you're using k8s 1.21, but we won't cut any extra release for v2.8 to backport the fix as this version has 1,5 years and k8s 1.21 is out of support too. The only solution to the bug of this feature is using an updated version of KEDA.

I also saw the design of autoscaling.keda.sh/paused, can it solve my problem?

Do you mean to modify the current instances of a pod manually? yeah, probably it can. I use the pause feature when I want to disable or modify the amount of pods of a workload during manual operations

[LY-today]

@tomkerkhove
Haha, but I tested autoscaling.keda.sh/paused: 'true' on 2.8.2 and found that manually adjusting replicas has no effect. Is there a design document that I can refer to? I really started to suspect that it was my operation problem. The official documentation does not seem to open paused to users.

[JorTurFer]

the annotation autoscaling.keda.sh/paused was introduced in KEDA v2.12 so it won't work in your case. You can check the feature for the v2.8 here -> https://keda.sh/docs/2.8/concepts/scaling-deployments/#pause-autoscaling

In your case, you have to use autoscaling.keda.sh/paused-replicas: "X" where X is the amount of instances that you want.

Manual changes over the replica count directly in the workload will always be overridden. If you want to change the amount of replicas, you have to update the annotation with the required value