-
Notifications
You must be signed in to change notification settings - Fork 7
Use_case_Config_policy
Kafka brokers support pluggable policy plugins for topic creation and entity configuration. Unfortunately these are not sufficiently powerful for many real-world use cases. For example, although you can apply your policy at topic creation, there is no policy for adding partitions to an existing topic. This makes it impossible to implement reasonable-sounding policies such as having a global limit on the total number of partitions and/or replicas in a cluster.
A proxy could intercept all the RPCs which affect topics:
- CreateTopics
- CreatePartitions
- (Incremental)AlterConfigs
- AlterPartitionReassignments
and reject those requests which don't conform to the policy.
Within a Kafka broker, policies are applied after authorization, which means that you cannot infer the existence of a topic which you are not authorized to describe by, for example. A policy-enforcing proxy would need to avoid leaking this kind of topic existence information to a client.
Policies which depend on cluster-global state are tricky to implement correctly in a network of distributed proxy instances. For example, when trying to enforce a limit on the number of partitions in a deployment with >1 proxy instance then, because Create(Topics|Partitions)Requests can be sent to different proxy instances, it's possible to go over the limit by routing two requests to different instances. To avoid this you'd want to route the intercepted requests to a single proxy instance and serialize decision-making there. In other words, the proxy instances need to form an HA cluster, and use an inter-proxy routing mechanism.
Some policies might depend on the identity of the connected client. For example, user A has a limit of X partitions, but user B has a limit of Y partitions. Policies like this require the filter to understand client identity.