how to limit number of partitions assigned to consumer

[nitin302]

For Kafka, partitions provide a way for consumption to scale. Hence, it is necessary that consumers needs to scale horizontally as well.

1. Is there any way to limit number of partitions per consumer, so that we can add more consumers so that all partitions are assigned to one of the consumers (total consumers * partitions per consumer = total partitions of the topic)? How can I do this?

2. What should the configuration of the node in this case? What logic can be used for size a consumer for a given number of partitions?

[dim]

@nitin302 I suggest that you look into IBM/sarama#1099 which has been merged and released, as sarama-cluster is now deprecated. You can configure a custom https://godoc.org/github.com/Shopify/sarama#BalanceStrategy which can plan the partition allocation to consumers based on a consumer's UserData. Hope that helps!

[Shan4Kites]

@dim can you please add "deprecated" in the github description for this repo? it will help the new users.