Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(quota plugins): note on quotas for internal users #10803

Merged
merged 6 commits into from
Nov 9, 2024
Merged

docs(quota plugins): note on quotas for internal users #10803

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a3c9e5d
docs(quota plugins): note on quotas for internal users
PaulRMellor Nov 5, 2024
b71249b
docs(review): edits to dow from comments by PW
PaulRMellor Nov 5, 2024
5d5ed70
docs(review): edits to doc from review by PM
PaulRMellor Nov 5, 2024
985bc9c
docs(review ): edits to doc from comments by JS
PaulRMellor Nov 6, 2024
d090b87
docs(review): edits to doc from comments by JS and KL
PaulRMellor Nov 7, 2024
ffbe38d
docs(review): edits to doc from comments by JS and KL -- 02
PaulRMellor Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions documentation/modules/managing/proc-setting-broker-limits.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@
This procedure describes how to set throughput and storage limits on brokers in your Kafka cluster.
Enable a quota plugin and configure limits using `quotas` properties in the `Kafka` resource.

.Quota plugins

There are two types of quota plugins available:

* The `strimzi` type enables the _Strimzi Quotas_ plugin.
Expand Down Expand Up @@ -46,6 +44,16 @@ Limits can be overridden by xref:con-configuring-client-quotas-str[user-specific
* CPU utilization limits for each client can be set as a percentage of the network threads and I/O threads on a per-broker basis.
* The number of concurrent partition creation and deletion operations (mutations) allowed per second can be set on a per-broker basis.

When using the default Kafka quotas plugin, the default quotas (if set) are applied to all users.
This includes internal users such as the Topic Operator and Cruise Control, which may impact their operations.
To avoid unduly limiting internal users, consider tuning the quotas effectively.

For example, a quota automatically applied to the Topic Operator by the Kafka quotas plugin could constrain the controller mutation rate, potentially throttling topic creation or deletion operations.
Therefore, it is important to understand the minimal quotas required by the Topic Operator to function correctly and explicitly set appropriate quotas to avoid such issues.
Monitoring relevant controller and broker metrics can help track and optimize the rate of operations on topics.
Cruise Control and its metrics reporter also require sufficient produce and fetch rates to conduct rebalances, depending on the scale and configuration of the Kafka cluster.
To prevent issues for Cruise Control, you might start with a rate of at least 1 KB/s for its producers and consumers in small clusters, such as three brokers with moderate traffic, and adjust as needed for larger or more active clusters.

.Prerequisites

* The Cluster Operator that manages the Kafka cluster is running.
Expand Down Expand Up @@ -104,9 +112,6 @@ spec:

. Apply the changes to the `Kafka` configuration.

NOTE: `minAvailableBytesPerVolume` and `minAvailableRatioPerVolume` are mutually exclusive.
This means that only one of these parameters should be configured.

NOTE: Additional options can be configured in the `spec.kafka.config` section.
The full list of supported options can be found in the https://github.com/strimzi/kafka-quotas-plugin?tab=readme-ov-file#properties-and-their-defaults[plugin documentation].

Expand Down
Loading