Azure Service Bus Scaler Authentication Issue with Workload Identity Description

[seifrajhi]

Report

I am encountering an issue while trying to use KEDA with Azure AD Workload Identity and Service Bus. I want to authenticate from my AKS cluster keda pod to the Service Bus, but I’m facing authentication-related errors.

KEDA ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: app-keda-scaled 
  namespace: gitops
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gitops-deploy
  triggers:
    - type: azure-servicebus
      metadata:
        namespace: "gitops-test"
        queueName: "gitops-test-queue"
        messageCount: "1"
      authenticationRef:
        name: app-keda

KEDA TriggerAuthentication:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: app-keda
spec:
  podIdentity:
    provider: azure-workload
    identityId: <my_identityId>

Where <my_identityId> is the Azure managed identity client ID.
I have assigned to this identity the role Azure Service Bus Data to resource service bus queue bus-demo-gitops/gitopsqueue

Expected Behavior

I should be able to authenticate from my AKS cluster to the Service Bus using Azure AD Workload Identity.

Actual Behavior

The Azure workload-identity token provider encounters an error.
The Service Bus scaler reports issues related to TokenCredential and missing TokenFilePath.

Steps to Reproduce the Problem

1. Deploy KEDA operator.
2. Set up trigger authentication with Azure Workload Identity and ScaledObject.
3. Override the credential.
4. Observe the error related to metrics retrieval and TokenCredential.

Logs from KEDA operator

keda-operator error logs:

2024-07-23T22:53:11Z	ERROR	azure_servicebus_scaler	error getting service bus entity length	{"type": "ScaledObject", "namespace": "gitops", "name": "gitops", "error": "sources must contain at least one TokenCredential"}
github.com/kedacore/keda/v2/pkg/scalers.(*azureServiceBusScaler).GetMetricsAndActivity
	/workspace/pkg/scalers/azure_servicebus_scaler.go:263
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsAndActivityForScaler
	/workspace/pkg/scaling/cache/scalers_cache.go:130
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics.func1
	/workspace/pkg/scaling/scale_handler.go:527
2024-07-23T22:53:11Z	ERROR	azure_servicebus_scaler	error starting azure workload-identity token provider	{"type": "ScaledObject", "namespace": "gitops", "name": "gitops", "error": "no token file specified. Check pod configuration or set TokenFilePath in the options"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
	/workspace/pkg/scaling/scale_handler.go:556
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
	/workspace/pkg/metricsservice/server.go:48
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler

the hpa output:

NAME               REFERENCE            TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-gitops  Deployment/gitops   <unknown>/1 (avg)   1         100       1          3h28m

Logs from keda-operator-metrics-apiserver

E0723 22:56:27.451588       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc000cd85e0)}: rpc error: code = Unknown desc = error when getting metric values metric:s0-azure-servicebus-gitopscqueue encountered error

KEDA Version

2.14.0

Kubernetes Version

1.28

Platform

Microsoft Azure

Scaler Details

Azure service bus

Anything else?

I have configured Azure managed identity, Federated Identity and Oidc for the AKS cluster as according to:
https://learn.microsoft.com/en-us/azure/aks/keda-workload-identity

[JorTurFer]

Hello,
Sorry for the slow response 😿
Could you double-check if KEDA operator pod has the needed env vars added by the workload identity webhook? These are the envs that you should have:

If you have followed the guide and they are not there, try restarting KEDA operator pod

[ppa007asr]

Hello,

We are encountering the same issue. KEDA operator is correctly configured but it seems the override of the client_id isnt working in the triggerAuthentication resource in combination with scaledJob. Please investigate

[JorTurFer]

Could you share KEDA operator logs, TriggerAuthentication, ScaledObject and KEDA version?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.