Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v2] Update scaled job #220

Merged
merged 10 commits into from
Aug 18, 2020
98 changes: 96 additions & 2 deletions content/docs/2.0/concepts/scaling-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ For example, if you wanted to use KEDA to run a job for each message that lands
1. When a message arrives on the queue, KEDA creates a job.
1. When the job starts running, it pulls *a single* message and processes it to completion.
1. As additional messages arrive, additional jobs are created. Each job processes a single message to completion.
1. Periodically remove completed/failed job by the `SuccessfulJobsHistoryLimit` and `FailedJobsHistoryLimit.`

## ScaledJob spec

Expand All @@ -35,11 +36,104 @@ spec:
template:
# describes the [job template](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)
pollingInterval: 30 # Optional. Default: 30 seconds
cooldownPeriod: 300 # Optional. Default: 300 seconds
minReplicaCount: 0 # Optional. Default: 0
successfulJobsHistoryLimit: 5 # Optional. Default: 100. How many completed jobs should be kept.
failedJobsHistoryLimit: 5 # Optional. Default: 100. How many failed jobs should be kept.
maxReplicaCount: 100 # Optional. Default: 100
triggers:
# {list of triggers to create jobs}
```

You can find all supported triggers [here](../scalers).

## Details

```yaml
jobTargetRef:
parallelism: 1 # Max number of desired instances ([docs](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism))
completions: 1 # Desired number of successfully finished instances ([docs](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism))
activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6
```

The optional configuration parameter. Currently not implemented. It is going to be supported.

```yaml
pollingInterval: 30 # Optional. Default: 30 seconds
```

This is the interval to check each trigger on. By default, KEDA will check each trigger source on every ScaledJob every 30 seconds.


```yaml
successfulJobsHistoryLimit: 5 # Optional. Default: 100. How many completed jobs should be kept.
failedJobsHistoryLimit: 5 # Optional. Default: 100. How many failed jobs should be kept.
```

The `successfulJobsHistoryLimit` and `failedJobsHistoryLimit` fields are optional. These fields specify how many completed and failed jobs should be kept. By default, they are set to 100.

This concept is similar to [Jobs History Limits](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#jobs-history-limits) allowing you to learn what the outcome of your jobs are.

The actual number of jobs could exceed the limit in a short time. However, it is going to resolve in the cleanup period. Currently, the cleanup period is the same as the Polling interval.

```yaml
maxReplicaCount: 100 # Optional. Default: 100
```

The max number of pods that is created within a single polling period. If there are running jobs, the number of running jobs will be deducted. This table is an example of the scaling logic.

| Queue Length | maxReplicaCount | TargetAvarageValue | RunningJobCount | Number of the Scale |
tomkerkhove marked this conversation as resolved.
Show resolved Hide resolved
| ------- | ------ | ------- | ------ | ----- |
| 10 | 3 | 1 | 0 | 3 |
| 10 | 3 | 2 | 0 | 3 |
| 10 | 3 | 1 | 1 | 2 |
| 10 | 100 | 1 | 0 | 10 |
| 4 | 3 | 5 | 0 | 1 |

* **Queue Length:** The number of the length of the queue.
* **Target Average Value:** How many pods do they have in a Job.
* **Running Job Count:** How many jobs are running.
* **Number of the Scale:** The number of the job that is created.

# Sample


```yaml
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-consumer
data:
RabbitMqHost: <omitted>
LocalHost: <omitted>
---
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: rabbitmq-consumer
namespace: default
spec:
jobTargetRef:
template:
spec:
containers:
- name: rabbitmq-client
image: tsuyoshiushio/rabbitmq-client:dev3
imagePullPolicy: Always
command: ["receive", "amqp://user:[email protected]:5672", "job"]
envFrom:
- secretRef:
name: rabbitmq-consumer
restartPolicy: Never
backoffLimit: 4
pollingInterval: 10 # Optional. Default: 30 seconds
maxReplicaCount: 30 # Optional. Default: 100
successfulJobsHistoryLimit: 3 # Optional. Default: 100. How many completed jobs should be kept.
failedJobsHistoryLimit: 2 # Optional. Default: 100. How many failed jobs should be kept.
triggers:
- type: rabbitmq
metadata:
queueName: hello
host: RabbitMqHost
localhost: LocalHost
queueLength : '5'
```