Azure Storage Queue scaling up to invisible messages

[jellevankerk]

Report

I use the Azure Storage Queue scaler for processes that need to run more that 5 minutes, which means that queue can look 'empty' (invisible messages) for a long time. with the fix from #4003, keda keeps scaling up pods while the queue being 'empty'.

Expected Behavior

keda to only scale down based on the invisible + visible messages and scale up based on only the visible messages

Actual Behavior

keda scales down based on the invisible + visible messages and scale up based on invisible + visible messages

Steps to Reproduce the Problem

1. Create a scaledjob with a storage queue as a trigger and pollingInterval 30
2. Send a message in the storage queue

Logs from KEDA operator

example

KEDA Version

2.10.0

Kubernetes Version

1.24

Platform

Microsoft Azure

Scaler Details

Azure Storage Queue

Anything else?

No response

[JorTurFer]

Hi,
But in the other case, you'd remove the instances processing messages yet, I mean, now with that fix the HPA controller doesn't kill pods if they are still processing messages. How were you ensuring that the HPA controller didn't kill the extra pods before the change?

[tomkerkhove]

He is using scaled job so HPA is not applicable here and jobs will just run.

[tomkerkhove]

I get why this is not ideal for your scenario, but I'm not sure I agree on your expected behavior though. Either we incorporate invisible+visible for both scale out/in, or we incorporate only visible; but not a combination based on scale in or out.

[tomkerkhove]

Created #4478 which is related

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.