More flexible Kafka topic configuration

[loicmathieu]

Proposal

Today, Kafka scaler only allow to scale based on the lag of one topic.

A lot of streaming applications uses multiple topics, using the Kafka scaler can be cumbersome if using tens of topics, or a regex of topics.

Multiple improvements can be added if the underlying Kafka client support them to configure multiple topics wich lags will be sumed:

• Configuring a list of topics
• Configuring a topic regex
• Taking into accound the lag of all topics of the configured consumer group

Use-Case

No response

Anything else?

No response

[loicmathieu]

Taking into accound the lag of all topics of the configured consumer group

Just seen this one is handled here : #2409, great ;)
But I still think it would be great to have the other capabilities.

[tomkerkhove]

• Configuring a list of topics
• Configuring a topic regex

Our current approach for all scalers is to add multiple triggers in your ScaledObject/Job. This should work for your scenario as well given it starts scaling as soon as one of them meets the criteria.

[loicmathieu]

@tomkerkhove some typical Kafka deployment involves multiple topic, for eg one topic by country. So you usually use a regex. And configuring hundreds of scaler is not very practical.

Moreover, it's different to have 100 scaler with a max lag of 100 and 1 scaler with a global lag of 10000, the scaler behaviour will not be the same.

[loicmathieu]

I agree that a list of topic can be replaced by multiple scalers but not the usage of a regex (moreover, regex target topics can be un-known at build time).

[zroubalik]

Regex is not a bad idea imho (btw this is implemented for RabbitMQ scaler).

@loicmathieu would you be interested to draft some POC?

[loicmathieu]

@zroubalik well I would love to but I'm pretty new to Go and the Sarama client didn't allow pattern based consumption OOTB so it may not be a trivial implementation. If nobody else wanted to work on it I can give it a try but I'm pretty busy these days so it will not be soon.

[zroubalik]

@loicmathieu ok, so let's keep this open for the community.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.