Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce BrokerSetAware Distribution Goals #2027

Open
mohitpali opened this issue Jun 19, 2023 · 0 comments
Open

Introduce BrokerSetAware Distribution Goals #2027

mohitpali opened this issue Jun 19, 2023 · 0 comments
Labels
functionality A feature request.

Comments

@mohitpali
Copy link
Contributor

Problem

BrokerSetAwareGoal was added as part of this PR. This Goal ensured that a topic remains within a set of brokers within the Kafka Cluster. A Set of Brokers within a Kafka Cluster is called a BrokerSet.
This Goal is a Hard Goal and one can not configure Distribution Goals along with BrokerSetAwareGoal. The distribution soft goals like ReplicaCapacityDistributionGoal/CPUCapacityDistributionGoal etc. currently are based out of average resource usage and help in proactive replica and leadership movements to keep the distribution of resources even across the Cluster.
Since BrokerSet awareness requires replica movements within the boundary of a BrokerSet, it should also do even distribution within a BrokerSet and not within the whole Cluster.
e.g. One BrokerSet B1 can have high usage and other BrokerSet B2 can have low usage. If B1 has an imbalance, we would want to even out resource usage using soft goals within the BrokerSet.

Approach

We would want to make distribution goals even out resource usage within a brokerSet. If BrokerSetAwareGoal is configured, then the CPU/NW/Disk averages should be individually calculated for each brokerSet.
We would want to modify the existing Distribution goals to generate broker set aware proposals that do even distribution without waiting for thresholds to reach.

@CCisGG CCisGG added the functionality A feature request. label Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
functionality A feature request.
Projects
None yet
Development

No branches or pull requests

2 participants