Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically pause downscaling based on defined metrics #5166

Open
spereverziev opened this issue Nov 7, 2023 · 5 comments
Open

Automatically pause downscaling based on defined metrics #5166

spereverziev opened this issue Nov 7, 2023 · 5 comments
Labels
feature-request All issues for new features that have not been committed to needs-discussion stale-bot-ignore All issues that should not be automatically closed by our stale bot

Comments

@spereverziev
Copy link

spereverziev commented Nov 7, 2023

Proposal

Dynamically pause autoscaling based on condition. E.g pause when the metric requests pers seconds is less then X

Use-Case

I have a service A that calls service B. When A has an outage it stop sending requests to B and therefore B starts to downscaling all the way to min. When A comes back online it starts storming B that doesn't have enough replicas to handle the load which can take it down or significantly increase the time to recover.

So I want to be able to pause downscaling of B when service A has an outage. I can define this by using custom metrics query like rps of B dropped below X.

Is this a feature you are interested in implementing yourself?

No

Anything else?

No response

@spereverziev spereverziev added feature-request All issues for new features that have not been committed to needs-discussion labels Nov 7, 2023
@SpiritZhou
Copy link
Contributor

Could you provide a more detailed use case? In my opinion, if the metric requests per second are less than X, the pods should be scaled down, right? Alternatively, if the metric has some extreme value, you can use maxReplicaCount.

@spereverziev
Copy link
Author

So the use case is the following. I have a service A that calls service B. When A has an outage it stop sending requests to B and therefore B starts to downscaling all the way to min. When A comes back online it starts storming B that doesn't have enough replicas to handle the load which can take it down or significantly increase the time to recover.

So I want to be able to pause downscaling of B when service A has an outage. I can define this by using custom metrics query like rps of B dropped bellow X.

I don't see how maxReplicasCount can help in my use case

@spereverziev
Copy link
Author

@SpiritZhou I added more context ^, thanks

@spereverziev
Copy link
Author

This is a very common use case that happens almost every month at my company

@spereverziev spereverziev changed the title Automatically pause autoscaling based on defined metrics Automatically pause downscaling based on defined metrics Nov 27, 2023
Copy link

stale bot commented Jan 27, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Jan 27, 2024
@zroubalik zroubalik added the stale-bot-ignore All issues that should not be automatically closed by our stale bot label Jan 29, 2024
@stale stale bot removed the stale All issues that are marked as stale due to inactivity label Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request All issues for new features that have not been committed to needs-discussion stale-bot-ignore All issues that should not be automatically closed by our stale bot
Projects
Status: To Triage
Development

No branches or pull requests

3 participants