From ae95490f57932ac182494c7e4b23da298a9928d2 Mon Sep 17 00:00:00 2001 From: Ram Cohen Date: Wed, 16 Feb 2022 16:24:03 +0200 Subject: [PATCH 1/7] Added documentation for the new GCP storage scaler Signed-off-by: Ram Cohen --- content/docs/2.7/scalers/gcp-pub-sub.md | 2 +- content/docs/2.7/scalers/gcp-storage.md | 148 ++++++++++++++++++++++++ 2 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 content/docs/2.7/scalers/gcp-storage.md diff --git a/content/docs/2.7/scalers/gcp-pub-sub.md b/content/docs/2.7/scalers/gcp-pub-sub.md index 004f6770c..8a1650e66 100644 --- a/content/docs/2.7/scalers/gcp-pub-sub.md +++ b/content/docs/2.7/scalers/gcp-pub-sub.md @@ -95,7 +95,7 @@ spec: **Identity based authentication:** -You can also use `TriggerAuthentication` CRD to configure the authentication using the associated service account of the running machine in Google Cloud. You only need to create a `TriggerAuthentication` as this example, and reference it in the `ScaledObject`. `ClusterTriggerAuthentication` can also be used if you pretend to use it globally in your cluster. +You can also use `TriggerAuthentication` CRD to configure the authentication using the associated service account of the running machine in Google Cloud. You only need to create a `TriggerAuthentication` as this example, and reference it in the `ScaledObject`. `ClusterTriggerAuthentication` can also be used if you intend to use it globally in your cluster. ### Example using TriggerAuthentication with GCP Identity diff --git a/content/docs/2.7/scalers/gcp-storage.md b/content/docs/2.7/scalers/gcp-storage.md new file mode 100644 index 000000000..40c9713e7 --- /dev/null +++ b/content/docs/2.7/scalers/gcp-storage.md @@ -0,0 +1,148 @@ ++++ +title = "Google Cloud Platform Storage" +availability = "2.7+" +maintainer = "Community" +description = "Scale applications based on the count of objects in a given GCP Storage bucket." +layout = "scaler" +go_file = "gcp_storage_scaler" ++++ + +### Trigger Specification + +This specification describes the `gcp-storage` trigger for GCP Storage. It scales based on the count of objects in a given GCS bucket and assumes the worker is responsible for clearing the bucket by deleting/moving the objects once their processing is complete. + +```yaml +triggers: +- type: gcp-storage + metadata: + bucketName: test-bucket + targetObjectCount: '100' + maxBucketItemsToScan: '1000' +``` + +**Parameter list:** + +- `bucketName` - Name of the bucket in GCP Storage. +- `targetObjectCount` - Average target value to trigger scaling actions. (Default: `100`, Optional) +- `maxBucketItemsToScan` - When to stop counting how many objects are in the bucket. (Default: `1000`, Optional) +As counting the number of objects involves iterating over their metadata it is advised to set this number to the value of `targetObjectCount` * `maxReplicaCount`. + +You can also optionally assign a name to the metric using the `metricName` value. If not specified, the `metricName` will be generated automatically based on the trigger index and `bucketName`. For example: **s0-gcp-storage-bucketName**. The value will be prefixed with `s{triggerIndex}-gcp-storage-`. + +You can provide in the metadata either `credentialsFromEnv` or `credentialsFromEnvFile`. +- `credentialsFromEnv` - Set to the name of the environment variable that holds the credential information. +- `credentialsFromEnvFile` - Set to the name of a json file that holds the credential information. + +### Authentication Parameters +You can use `TriggerAuthentication` CRD to configure the authenticate by providing the service account credentials in JSON. + +**Credential based authentication:** + +- `GoogleApplicationCredentials` - Service account credentials in JSON. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: gcp-storage-scaledobject + namespace: keda-gcp-storage-test +spec: + scaleTargetRef: + name: keda-gcp-storage-go + triggers: + - type: gcp-storage + metadata: + bucketName: "Transactions" + targetObjectCount: "5" + credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON +``` + +### Example using TriggerAuthentication + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-gcp-credentials +spec: + secretTargetRef: + - parameter: GoogleApplicationCredentials + name: gcp-storage-secret # Required. Refers to the name of the secret + key: GOOGLE_APPLICATION_CREDENTIALS_JSON # Required. +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: gcp-storage-scaledobject +spec: + scaleTargetRef: + name: keda-gcp-storage-go + triggers: + - type: gcp-storage + authenticationRef: + name: keda-trigger-auth-gcp-credentials + metadata: + bucketName: "Transactions" + targetObjectCount: "5" +``` + +**Identity based authentication:** + +You can also use `TriggerAuthentication` CRD to configure the authentication using the associated service account of the running machine in Google Cloud. You only need to create a `TriggerAuthentication` as this example, and reference it in the `ScaledObject`. `ClusterTriggerAuthentication` can also be used if you intend to use it globally in your cluster. + +### Example using TriggerAuthentication with GCP Identity + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-gcp-credentials +spec: + podIdentity: + provider: gcp +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: gcp-storage-scaledobject +spec: + scaleTargetRef: + name: keda-gcp-storage-go + triggers: + - type: gcp-storage + authenticationRef: + name: keda-trigger-auth-gcp-credentials + metadata: + bucketName: "Transactions" + targetObjectCount: "5" +``` + +## Example using ClusterTriggerAuthentication with GCP Identity + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ClusterTriggerAuthentication +metadata: + name: keda-clustertrigger-auth-gcp-credentials +spec: + podIdentity: + provider: gcp +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: gcp-storage-scaledobject +spec: + scaleTargetRef: + name: keda-gcp-storage-go + triggers: + - type: gcp-storage + authenticationRef: + name: keda-clustertrigger-auth-gcp-credentials + kind: ClusterTriggerAuthentication + metadata: + bucketName: "Transactions" + targetObjectCount: "5" +``` \ No newline at end of file From 8cdb9220a8bc8f64235c05ac4ee4bb940cce85cc Mon Sep 17 00:00:00 2001 From: Ram Cohen Date: Thu, 17 Feb 2022 14:34:48 +0200 Subject: [PATCH 2/7] Review fixes Signed-off-by: Ram Cohen --- content/docs/2.7/scalers/gcp-storage.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/content/docs/2.7/scalers/gcp-storage.md b/content/docs/2.7/scalers/gcp-storage.md index 40c9713e7..3ef63e216 100644 --- a/content/docs/2.7/scalers/gcp-storage.md +++ b/content/docs/2.7/scalers/gcp-storage.md @@ -18,6 +18,8 @@ triggers: bucketName: test-bucket targetObjectCount: '100' maxBucketItemsToScan: '1000' + credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON # Optional + credentialsFromEnvFile: GOOGLE_APPLICATION_CREDENTIALS_JSON # Optional ``` **Parameter list:** @@ -40,7 +42,11 @@ You can use `TriggerAuthentication` CRD to configure the authenticate by providi - `GoogleApplicationCredentials` - Service account credentials in JSON. -### Example +**Identity based authentication:** + +You can also use `TriggerAuthentication` CRD to configure the authentication using the associated service account of the running machine in Google Cloud. You only need to create a `TriggerAuthentication` as this example, and reference it in the `ScaledObject`. `ClusterTriggerAuthentication` can also be used if you intend to use it globally in your cluster. + +### Examples ```yaml apiVersion: keda.sh/v1alpha1 @@ -59,7 +65,7 @@ spec: credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON ``` -### Example using TriggerAuthentication +### Using TriggerAuthentication ```yaml apiVersion: keda.sh/v1alpha1 @@ -88,10 +94,6 @@ spec: targetObjectCount: "5" ``` -**Identity based authentication:** - -You can also use `TriggerAuthentication` CRD to configure the authentication using the associated service account of the running machine in Google Cloud. You only need to create a `TriggerAuthentication` as this example, and reference it in the `ScaledObject`. `ClusterTriggerAuthentication` can also be used if you intend to use it globally in your cluster. - ### Example using TriggerAuthentication with GCP Identity ```yaml From d2785df87cb009e98a16d4e544ce62b87e7f1362 Mon Sep 17 00:00:00 2001 From: Ram Cohen Date: Sun, 20 Feb 2022 11:59:56 +0200 Subject: [PATCH 3/7] Update .gitignore Signed-off-by: Ram Cohen --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 7e9c13bf0..93e3d099e 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,7 @@ package-lock.json .idea resources/ .DS_Store +.hugo_build.lock # Link checker artifacts bin/ From b43dea84ddb50b54d8d592c86fe077c0e86bcaf5 Mon Sep 17 00:00:00 2001 From: Ram Cohen Date: Fri, 18 Feb 2022 17:01:10 +0200 Subject: [PATCH 4/7] Apply suggestions from code review Co-authored-by: Tom Kerkhove Signed-off-by: Ram Cohen --- content/docs/2.7/scalers/gcp-storage.md | 32 ++----------------------- 1 file changed, 2 insertions(+), 30 deletions(-) diff --git a/content/docs/2.7/scalers/gcp-storage.md b/content/docs/2.7/scalers/gcp-storage.md index 3ef63e216..c35681cd2 100644 --- a/content/docs/2.7/scalers/gcp-storage.md +++ b/content/docs/2.7/scalers/gcp-storage.md @@ -65,7 +65,7 @@ spec: credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON ``` -### Using TriggerAuthentication +#### Use TriggerAuthentication with Kubernetes secret ```yaml apiVersion: keda.sh/v1alpha1 @@ -94,7 +94,7 @@ spec: targetObjectCount: "5" ``` -### Example using TriggerAuthentication with GCP Identity +#### Use TriggerAuthentication with GCP Identity ```yaml apiVersion: keda.sh/v1alpha1 @@ -119,32 +119,4 @@ spec: metadata: bucketName: "Transactions" targetObjectCount: "5" -``` - -## Example using ClusterTriggerAuthentication with GCP Identity - -```yaml -apiVersion: keda.sh/v1alpha1 -kind: ClusterTriggerAuthentication -metadata: - name: keda-clustertrigger-auth-gcp-credentials -spec: - podIdentity: - provider: gcp ---- -apiVersion: keda.sh/v1alpha1 -kind: ScaledObject -metadata: - name: gcp-storage-scaledobject -spec: - scaleTargetRef: - name: keda-gcp-storage-go - triggers: - - type: gcp-storage - authenticationRef: - name: keda-clustertrigger-auth-gcp-credentials - kind: ClusterTriggerAuthentication - metadata: - bucketName: "Transactions" - targetObjectCount: "5" ``` \ No newline at end of file From 7d8748a88eafd8e915d0f2f55f1376045b9b0090 Mon Sep 17 00:00:00 2001 From: Ram Cohen Date: Tue, 1 Mar 2022 11:57:50 +0200 Subject: [PATCH 5/7] Applied suggestion to initial paragraph Signed-off-by: Ram Cohen --- content/docs/2.7/scalers/gcp-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/2.7/scalers/gcp-storage.md b/content/docs/2.7/scalers/gcp-storage.md index c35681cd2..9890ee5ee 100644 --- a/content/docs/2.7/scalers/gcp-storage.md +++ b/content/docs/2.7/scalers/gcp-storage.md @@ -9,7 +9,7 @@ go_file = "gcp_storage_scaler" ### Trigger Specification -This specification describes the `gcp-storage` trigger for GCP Storage. It scales based on the count of objects in a given GCS bucket and assumes the worker is responsible for clearing the bucket by deleting/moving the objects once their processing is complete. +This specification describes the `gcp-storage` scaler, which scales Kubernetes workloads based on the count of objects in a given Google Cloud Storage (GCS) bucket. This scaler assumes the worker, when run, will process and clear the bucket by deleting/moving objects therein. ```yaml triggers: From 2ba6173d9d4707113f34159f3b8fc10959c755d4 Mon Sep 17 00:00:00 2001 From: Ram Cohen Date: Tue, 1 Mar 2022 12:58:07 +0200 Subject: [PATCH 6/7] Remove metricName option Signed-off-by: Ram Cohen --- content/docs/2.7/scalers/gcp-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/2.7/scalers/gcp-storage.md b/content/docs/2.7/scalers/gcp-storage.md index 9890ee5ee..68ec30b73 100644 --- a/content/docs/2.7/scalers/gcp-storage.md +++ b/content/docs/2.7/scalers/gcp-storage.md @@ -29,7 +29,7 @@ triggers: - `maxBucketItemsToScan` - When to stop counting how many objects are in the bucket. (Default: `1000`, Optional) As counting the number of objects involves iterating over their metadata it is advised to set this number to the value of `targetObjectCount` * `maxReplicaCount`. -You can also optionally assign a name to the metric using the `metricName` value. If not specified, the `metricName` will be generated automatically based on the trigger index and `bucketName`. For example: **s0-gcp-storage-bucketName**. The value will be prefixed with `s{triggerIndex}-gcp-storage-`. +The metric name will be generated automatically based on the trigger index and `bucketName`, for example: **s0-gcp-storage-bucketName**. You can provide in the metadata either `credentialsFromEnv` or `credentialsFromEnvFile`. - `credentialsFromEnv` - Set to the name of the environment variable that holds the credential information. From 291f5b1821bbbd82f3f99037affeb79394dd15de Mon Sep 17 00:00:00 2001 From: Ram Cohen Date: Thu, 3 Mar 2022 09:57:15 +0200 Subject: [PATCH 7/7] Use GCS instead of GCP storage Signed-off-by: Ram Cohen --- content/docs/2.7/scalers/gcp-storage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/2.7/scalers/gcp-storage.md b/content/docs/2.7/scalers/gcp-storage.md index 68ec30b73..0e016d080 100644 --- a/content/docs/2.7/scalers/gcp-storage.md +++ b/content/docs/2.7/scalers/gcp-storage.md @@ -2,7 +2,7 @@ title = "Google Cloud Platform Storage" availability = "2.7+" maintainer = "Community" -description = "Scale applications based on the count of objects in a given GCP Storage bucket." +description = "Scale applications based on the count of objects in a given Google Cloud Storage (GCS) bucket." layout = "scaler" go_file = "gcp_storage_scaler" +++ @@ -24,7 +24,7 @@ triggers: **Parameter list:** -- `bucketName` - Name of the bucket in GCP Storage. +- `bucketName` - Name of the bucket in GCS. - `targetObjectCount` - Average target value to trigger scaling actions. (Default: `100`, Optional) - `maxBucketItemsToScan` - When to stop counting how many objects are in the bucket. (Default: `1000`, Optional) As counting the number of objects involves iterating over their metadata it is advised to set this number to the value of `targetObjectCount` * `maxReplicaCount`.