Scaler for GCP PubSub

[patnaikshekhar]

Added new scaler for GCP PubSub. It uses the stackdriver api to determine the number of messages in the pubsub subscription

[patnaikshekhar]

This is for issue 141

[patnaikshekhar]

@ahmelsayed could you please review this change? It's a scaler for GCP pub sub and uses the Stackdriver API to fin the number of pending messages in a subscription.

[ahmelsayed]

@patnaikshekhar very sorry about the delay for your 2 PRs. I wanted to learn a bit more about both GCP PubSub and AAD Pods and get to try it before merging but kept getting randomized. I'll finish my review by Monday.

[patnaikshekhar]

No problem. I'm in the process of creating samples so that might help! :)

[ahmelsayed]

@patnaikshekhar the code looks fine to me. however trying it I'm getting this error:

Error getting scale decision: Could not find stackdriver metric with filter metric.type=\"pubsub.googleapis.com/subscription/num_undelivered_messages\" AND resource.labels.subscription_id=\"projects/swift-kiln-183720/subscriptions/foo\""

I have a simple nodejs producer

pubsub
    .topic('projects/swift-kiln-183720/topics/stuff')
    .publisher
    .publish(Buffer.from(new Date() + ''))
    .then(messageId => {
        console.log(`Message ${messageId} published.`);
    })
    .catch(err => {
        console.error('ERROR:', err);
    })

and consumer

const subscription = pubsub.subscription('projects/swift-kiln-183720/subscriptions/foo')
subscription.on(`message`, msg => {
    console.log(msg)
    msg.ack()
})

I can see my producer and consumer working. This is my trigger in the scaledObject

  - type: gcp-pubsub
    metadata:
      subscriptionSize: "5"
      subscriptionName: "projects/swift-kiln-183720/subscriptions/foo"
      credentials: googcreds

I'm just learning about PubSub so excuse me if it's a very dummy question.

[patnaikshekhar]

@ahmelsayed have you given the service account access to view stackdriver logs?

gcloud projects add-iam-policy-binding $PROJECT_ID
--member $SERVICE_ACCOUNT_NAME
--role roles/monitoring.viewer

Additionally the subscription name is just the name of the subscription without the project ID. I'm automatically picking up the project ID from the JSON of the service account. (Maybe I shouldn't be?)

kubectl create secret generic pubsub-secret
--from-file=GOOGLE_APPLICATION_CREDENTIALS_JSON=./sa.json
--from-literal=PROJECT_ID=$PROJECT_ID
-n keda-pubsub-test

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: pubsub-scaledobject
  namespace: keda-pubsub-test
  labels:
    deploymentName: keda-pubsub-go
spec:
  scaleTargetRef:
    deploymentName: keda-pubsub-go
  triggers:
  - type: gcp-pubsub
    metadata:
      subscriptionSize: "5"
      subscriptionName: "sub1" # Required 
      credentials: GOOGLE_APPLICATION_CREDENTIALS_JSON # Required
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keda-pubsub-go
  namespace: keda-pubsub-test
spec:
  selector:
    matchLabels:
      service: keda-pubsub-go
  replicas: 1
  template:
    metadata:
      labels:
        service: keda-pubsub-go
    spec:
      containers:
      - image: patnaikshekhar/keda-pubsub-sample:1
        name: consumer
        env:
        - name: SUBSCRIPTION_NAME
          value: "sub1"
        - name: GOOGLE_APPLICATION_CREDENTIALS_JSON
          valueFrom:
            secretKeyRef:
              name: pubsub-secret
              key: GOOGLE_APPLICATION_CREDENTIALS_JSON

I'm out of office today.. but I'll publish the sample tomorrow containing all the steps required

[patnaikshekhar]

@ahmelsayed here is a link to the WIP tutorial for this scaler.

[ahmelsayed]

Followed your tutorial and everything worked for me!

Thank you very much @patnaikshekhar for your contribution and sorry it took me a while to get to the PR.

[ahmelsayed]

@patnaikshekhar I think it would be nice if we add your repo (https://github.com/patnaikshekhar/KEDA-GCP-PubSub-Sample) to the kedacore org as another sample repo like the ones here. I can fork it into the kedacore org, or create a new one. Not sure what you'd prefer.

/cc @jeffhollan