docs: update Apache Kafka Scaler Doc for multi topic lag (kedacore#613)

Signed-off-by: Jinli Liang <[email protected]>
Signed-off-by: Daniel Yavorovych <[email protected]>

content/docs/2.6/scalers/apache-kafka.md

@@ -8,8 +8,11 @@ go_file = "kafka_scaler"
 +++
 
 > **Notice:**
-> - By default, the number of replicas will not exceed the number of partitions on a topic.
- That is, if `maxReplicaCount` is set more than number of partitions, the scaler won't scale up to target maxReplicaCount. See `allowIdleConsumers` below to disable this default behavior.
+> - By default, the number of replicas will not exceed:
+>   - The number of partitions on a topic when a topic is specified;
+>   - The number of partitions of *all topics* in the consumer group when no topic is specified;
+>
+>   That is, if `maxReplicaCount` is set more than number of partitions, the scaler won't scale up to target maxReplicaCount. See `allowIdleConsumers` below to disable this default behavior.
 > - This is so because if there are more number of consumers than the number of partitions in a topic, then extra consumer will have to sit idle.
 
 ### Trigger Specification
@@ -33,13 +36,30 @@ triggers:
 
 - `bootstrapServers` - Comma separated list of Kafka brokers "hostname:port" to connect to for bootstrap.
 - `consumerGroup` - Name of the consumer group used for checking the offset on the topic and processing the related lag.
-- `topic` - Name of the topic on which processing the offset lag.
+- `topic` - Name of the topic on which processing the offset lag. (Optional, see note below)
 - `lagThreshold` - Average target value to trigger scaling actions. (Default: `5`, Optional)
 - `offsetResetPolicy` - The offset reset policy for the consumer. (Values: `latest`, `earliest`, Default: `latest`, Optional)
 - `allowIdleConsumers` - When set to `true`, the number of replicas can exceed the number of
 partitions on a topic, allowing for idle consumers. (Default: `false`, Optional)
 - `version` - Version of your Kafka brokers. See [samara](https://github.com/Shopify/sarama) version (Default: `1.0.0`, Optional)
 
+> **Note:**
+>
+> When `topic` is unspecified, total offset lag will be calculated with all topics within the consumer group.
+> - When there are **active** consumer instances, _all topics_ includes:
+>   - Topics the consumer is *currently* subscribing to;
+>   - Topics that the consumer group *had prior commit history* (up to retention period for `__consumer_offset`, default to 7 days, see [KIP-186](https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days));
+> - When there are **no active** consumer instances, _all topics_ only includes topics that the consumer group *had prior commit history*;
+> ---
+> An edge case exists where scaling could be **effectively disabled**:
+>    - Consumer never makes a commit (no record in `__consumer_offset`);
+>    - and `ScaledObject` had `minReplicaCount` as 0;
+>
+>   In such case, KEDA could scale the consumer down to 0 when there is no lag and won't be able scale up due to the topic could not be auto discovered.
+>
+> Fix for such case:
+>  - Set `minReplicaCount` > 0;
+>  - or use multiple triggers where one supplies `topic` to ensure lag for that topic will always be detected;
 ### Authentication Parameters
 
  You can use `TriggerAuthentication` CRD to configure the authenticate by providing `sasl`, `username` and `password`, in case your Kafka cluster has SASL authentication turned on. If TLS is required you should set `tls` to `enable`. If required for your Kafka configuration, you may also provide a `ca`, `cert` and `key`. `cert` and `key` must be specified together.