RabbitMQ ignores unacknowledged messages since >=2.0

[smoke]

Report

RabbitMQ scaler is using AMQP library and the method to get a total of messages returns only messages with ready status in queue.
https://github.com/streadway/amqp/blob/master/channel.go#L837

If the queue has messages with unacknowledged (because are processing in consumer) they will be ignored and the scaler can kill the pod that are processing messages making message returns to queue and create an infinite cycle :(

This was fixed for 1.x and this report closed #638
by introducing includeUnacked metadata option #700
that was later broken with >=2.0 with couple of PRs

Expected Behavior

Return total of messages in the queue

Actual Behavior

Return the total of ready messages and there is no way to switch to Total (Total = Ready + Unacked)

Steps to Reproduce the Problem

1. Create a queue
2. Publish 3 messages in the queue
3. Consume one message and do not confirm (ACK or NACK), the message will be in unacknowledged status.
4. Run Keda, the total of messages will be 2 (should be 3)

Logs from KEDA operator

example

KEDA Version

2.4.0

Kubernetes Version

1.21

Platform

Any

Scaler Details

RabbitMQ

Anything else?

No response

[coderanger]

The amqp protocol only exposes unacked messages, to include those you have to use the http mode which talks to the Management API instead.

[smoke]

@coderanger I have reverse engineered the code - regardless of the protocol they are not collected, this is something removed in other change sets :(

[coderanger]

https://github.com/kedacore/keda/blob/main/pkg/scalers/rabbitmq_scaler.go#L367 its right there :)

[smoke]

lol thank you @coderanger will give it a try and update the ticket accordingly!

[masterphenix]

Hello, I am having this same issue, both with HTTP and AMQP protocols
I am using Rabbitmq 3.9.11, and Keda 2.5.0
I use a TriggerAuthentication, and tried both AMQP and HTTP protocols
When creating a ScaledObject based on QueueLength, the metric shown in the HPA only takes into account ready messages, not the unacked ones.

[JorTurFer]

I've checked and can see that there is another property for messages_unacknowledged but is not used anywhere since v1.5 https://github.com/kedacore/keda/blob/main/pkg/scalers/rabbitmq_scaler.go#L79
It was used in v1.4 and maybe there is the problem https://github.com/kedacore/keda/blob/v1.4.0/pkg/scalers/rabbitmq_scaler.go#L178
WDYT @coderanger ? are they implicitly counted?

[JorTurFer]

Okey, digging into the returned json, I can see that there are 3 different keys, messages_ready, messages_ready and message_stats.publish. As I understand, message_stats.publish= messages_ready + messages_unacknowledged.
Am I right?
We are using message_stats.publish so if I'm right, the unacknowledged messages are already counted right now using http protocol

[bartbkr]

@JorTurFer Is the same counting mechanism used for useRegex: true? In other words, does it use message_stats.publish when protocol: http and mode: QueueLength? If the answer is yes, should this doc be ammended:

https://keda.sh/docs/2.6/scalers/rabbitmq-queue/

useRegex: "true" requires protocol http and ignores unacknowledged messages.

[JorTurFer]

I'm not sure @bartbkr. Depending on how it works, I'll update the docs according to the reality

[coderanger]

Not sure about the regex mode as I've never used it but it's pulling from the same "messages" field in the JSON so I would assume that's including messages of all states unless there is evidence to the contrary :) That field is defined as "the number of messages in the queue".

[JorTurFer]

so, message_stats.publish is the total amount of messages in the queue, including ready and unack, right? (ignore the regex part, you are right and using regex the json is the same)

[coderanger]

Yes, but we're not using that. We're using the "messages" key. We could switch to using the other struct but not sure it would change anything?

[JorTurFer]

yes, sorry. IDK why I was focus on that even thought I checked it, but I guess that messages is also the sum between ready and unack
Probably we should update the docs because, from them, it seems that unack are excluded

[bartbkr]

@JorTurFer Thank you and thanks for the clarification.

[masterphenix]

Hello,
Just tested with Curl on my 3.9.11 version, I don't get same results in messages and message_stats. At the time of the test, I have:

• 0 messages ready
• 247 messages unacked

This is the interesting parts of the json returned:

"message_stats":
	{"ack":594668,
	"ack_details":{"rate":12.4},	
	"deliver":844909,
	"deliver_details":{"rate":9.2},
	"deliver_get":844909,
	"deliver_get_details":{"rate":9.2},
	"deliver_no_ack":0,
	"deliver_no_ack_details":{"rate":0.0},
	"get":0,"get_details":{"rate":0.0},
	"get_empty":0,
	"get_empty_details":{"rate":0.0},
	"get_no_ack":0,
	"get_no_ack_details":{"rate":0.0},
	"publish":628756,
	"publish_details":{"rate":10.0},
	"redeliver":250032,
	"redeliver_details":{"rate":0.0}},

"messages":247,

[coderanger]

@masterphenix _stats is rolling counts since the node started. Useful for rate checks (which is why they are presented there) but not current sizes.

[masterphenix]

Ok @coderanger thank you for this explaination 👍