You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
BrokerSetAwareGoal was added as part of this PR. This Goal ensured that a topic remains within a set of brokers within the Kafka Cluster. A Set of Brokers within a Kafka Cluster is called a BrokerSet.
This Goal is a Hard Goal and one can not configure Distribution Goals along with BrokerSetAwareGoal. The distribution soft goals like ReplicaCapacityDistributionGoal/CPUCapacityDistributionGoal etc. currently are based out of average resource usage and help in proactive replica and leadership movements to keep the distribution of resources even across the Cluster.
Since BrokerSet awareness requires replica movements within the boundary of a BrokerSet, it should also do even distribution within a BrokerSet and not within the whole Cluster.
e.g. One BrokerSet B1 can have high usage and other BrokerSet B2 can have low usage. If B1 has an imbalance, we would want to even out resource usage using soft goals within the BrokerSet.
Approach
We would want to make distribution goals even out resource usage within a brokerSet. If BrokerSetAwareGoal is configured, then the CPU/NW/Disk averages should be individually calculated for each brokerSet.
We would want to modify the existing Distribution goals to generate broker set aware proposals that do even distribution without waiting for thresholds to reach.
The text was updated successfully, but these errors were encountered:
Problem
BrokerSetAwareGoal was added as part of this PR. This Goal ensured that a topic remains within a set of brokers within the Kafka Cluster. A Set of Brokers within a Kafka Cluster is called a BrokerSet.
This Goal is a Hard Goal and one can not configure Distribution Goals along with BrokerSetAwareGoal. The distribution soft goals like ReplicaCapacityDistributionGoal/CPUCapacityDistributionGoal etc. currently are based out of average resource usage and help in proactive replica and leadership movements to keep the distribution of resources even across the Cluster.
Since BrokerSet awareness requires replica movements within the boundary of a BrokerSet, it should also do even distribution within a BrokerSet and not within the whole Cluster.
e.g. One BrokerSet B1 can have high usage and other BrokerSet B2 can have low usage. If B1 has an imbalance, we would want to even out resource usage using soft goals within the BrokerSet.
Approach
We would want to make distribution goals even out resource usage within a brokerSet. If BrokerSetAwareGoal is configured, then the CPU/NW/Disk averages should be individually calculated for each brokerSet.
We would want to modify the existing Distribution goals to generate broker set aware proposals that do even distribution without waiting for thresholds to reach.
The text was updated successfully, but these errors were encountered: