-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Downscaling with Keda ScaledObjects not working #96
Comments
Hi @cecchcc could you share:
|
Hello @samuel-esp, here is the different informations
The information of Kubedownscaler
Here is a sample of the log concerning our deployment
Kube Downscaler does not give us more logs than that and it always say there is 1 replica even if there is more, for example here, we have 2 replicas for our backend-api but it still indicates 1. |
Hi @cecchcc thank you for your answer, from what I'm understanding you are trying to downscale the deployment in this time interval
keeping the annotation above means the Deployment will be excluded from downscaling, so you should delete that and replace with:
For Keda Scaled Object this annotation
Can you try to test this configuration? (changing the time interval of course to now) Looking from the time inside the logs, it is correct the workloads are not downscaled if you are currently targeting this time interval Mon-Fri 18:00-22:00 Europe/Paris
I see the concern for this log. Unfortunately the message is a "not elegant" way to say "Scaled Object is not downscaled yet", I will try to address this point with a new log message for keda in the next release |
Hi @samuel-esp , thank you for your answer. We tried with and without the
I don't understand why as we specified here the uptime of the application and not the downtime, so we should have 2 replicas between 18:00-22:00 and between 22:00 to 18:00 only 1 replica. But currently, we still have 2 no matter what the time is. |
Sorry, I just read the annotation in the wrong way (
|
You should be able to make it work using this configuration
The good news is that the behavior I wasn't expecting didn't happen
So it is sufficient to specify Let me know if it helped you! I'll try to clarify logs and behaviors in a PR for the next release. Thank you a lot for raising this question |
I tried to add the annotations on the deployment and the scaledobject but it is still not functionning well. In fact, the deployment scales down but Keda scales it up because it does not match its desired number of pods. Here are the logs and we noticed an error with the
|
Hello @samuel-esp , We do not have the error anymore when we use the v24.8.2 and it downscales the deployment but the downscaler/downtime-replicas: '1' is not applied as it scales down the number of replicas to 0. Do you have any idea why? |
Hi @cecchcc, it seems that you encountered the behavior i was suspecting. Then you shoudl wait until we add the official compatibility for "downscaler/downtime-replicas" inside ScaledObject annotation. Just give me some time, because I want to test it again, when I first tried to replicate your situation that behavior you are describing didn't happen. I just want to double check again |
I managed to replicate agian your situation, and I can confirm the behavior you are describing it's happening. Wait for the next release and I'll include the support for |
Hello @samuel-esp , I saw that there was a new release today. I tried it but we are getting this error now
Kube-downscaler can't downscale and upscale because of a read timeout. We did not have this issue before. Did you encounter this issue already? We added these 2 annotations on the
|
Hi @cecchcc are you running a managed Kubernetes (EKS, AKS, GKE) or self hosted? Are you encountering this error only for Scaled Object or also for other workloads? At first sight it seems the Kubernetes API Server is throttling |
This is the KubeDownscaler config
This is my log
|
We are using EKS. I tried using the downscaler with deployments and it is working fine
We only encounter the error with the |
Could you also share the configuration of KubeDownscaler deployment? How many workloads do you have inside your cluster that are in target of downscaling operation (more or less)? Are you using other workloads that needs to communicate a lot with the API Server? Also it would be great if you could replicate the issue inside another Cluster. @Fovty @JTaeuber are you able guys to test it as well? Inside my test Cluster I wasn't able to reproduce this, I'll try soon with another one |
We only have 1 workload, we are using mostly Here is the Kubedownscaler configuration
|
Sorry for asking many questions: could you provide also the version you are using for both Kubernetes and Keda? |
@JTaeuber @Fovty I saw PyKube was a little behind with dependencies so I opened caas-team/new-pykube#27 to bump dependencies there as well. If I don't manage to replicate the issue tomorrow as well, I'll try to build a custom image for kube-downscaler with the new pykube version and I'll give it to @cecchcc to test it. It may be some weird network issue where pykube and kube-downscaler dependencies need to be perfectly aligned |
The new release seems to work fine with older kubernetes version releases as well (just tested on 1.27 self managed). I will try to understand if it is EKS related @cecchcc another test you could do on your side is to try to replicate the behavior inside another test cluster if you have it at your disposal |
We are using Keda 2.15.1 and Kubernetes 1.30 |
@cecchcc can you join the slack in the docs? I'll give you some instruction to test a new image later this afternoon |
Issue
Hello,
We deployed py-kube-downscaler with Helm on our cluster and wanted to use it with Keda ScaledObjects. We annotated the ScaledObject with
downscaler/downtime-replicas
anddownscaler/uptime
and we also tried to use the annotationdownscaler/exclude: "true"
on the deployment like it is written on the doc.But it does not have any effect, the number of pods are not downscaling.
When deploying
py-kube-downscaler
we launched it with- '--include-resources=deployments,statefulsets,scaledobjects'
.Are we missing something?
The text was updated successfully, but these errors were encountered: