[ISSUE] Identity unable to refresh tokens

[Ben-Sheppard]

Related issues
https://jira.camunda.com/browse/SUPPORT-21543
SUPPORT-22007

Describe the issue:

When using the latest version of the helm charts Identity is unable to successfully use a refresh token to generate a new JWT. After further investigation this is linked to a recent Keycloak upgrade (REF).

We experienced the same issues with calls originating from different contexts (internal vs external), the resolution in this instance was to add KEYCLOAK_EXTRA_ARGS: "--hostname=localhost:18080 --hostname-strict-backchannel=false" to our Keycloak container. Applying the same logic to this issue here, using extraStartupArgs: "--hostname-url=https://keycloak.local.distro.ultrawombat.com/auth" forces the Keycloak instance to return the URLs as the public url and not the internal ([http://camunda-platform-keycloak:80](http://camunda-platform-keycloak/)) URL.

Applying the following configuration to the Keycloak container resolves the issue:

  extraStartupArgs: "--hostname-url=https://keycloak.local.distro.ultrawombat.com/auth"

Actual behavior:

When Identity tries to refresh the users access token an internal server error occurs and the Identity UI breaks requiring the user to clear session data and login again.

Expected behavior:

Identity should be able to successfully use a refresh token to obtain a new JWT for the user.

How to reproduce:

1. Use the values file attached
2. Login to Identity
3. Wait for 5 minutes (such that the JWT expires but the refresh token is valid)
4. Navigate to a different page in the Identity UI
5. Observe the error in the UI
6. Observe the logs on the Identity container suggesting an issuer validation problem

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

• Platform: Local KIND
• Helm CLI version: v3.14.4
• Chart version: Using main but also issue is in 10.2
• Values file:

global:
  multitenancy:
    enabled: true
    image:
      tag: latest
  identity:
    auth:
      publicIssuerUrl: "https://keycloak.local.distro.ultrawombat.com/auth/realms/camunda-platform"
      zeebe:
        existingSecret: "zeebeSecret"

identityPostgresql:
  enabled: true
  postgresql:
    enabled: true
identityKeycloak:
  auth:
    adminPassword: dummySecret1
    managementPassword: dummySecret2
  ingress:
    enabled: true
    hostname: keycloak.local.distro.ultrawombat.com
    extraTls:
      - hosts:
          - keycloak.local.distro.ultrawombat.com
        secretName: camunda-platform
  extraEnvVars:
    - name: KEYCLOAK_PROXY_ADDRESS_FORWARDING
      value: "true"
  postgresql:
    auth:
      password: dummySecret3

identity:
  image:
    tag: SNAPSHOT
  firstUser:
    enabled: true
    username: demo
    password: demo
    email: [email protected]
    firstName: Demo
    lastName: Demo

  auth:
    password: "identity"

  ingress:
    enabled: true
    host: identity.local.distro.ultrawombat.com
    tls:
      enabled: true
      secretName: "camunda-platform"

zeebe:
  retention:
    enabled: true
    minimumAge: 30d
    policyName: zeebe-record-retention-policy
  debug: true
  logLevel: debug
  javaOpts: >-
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=/usr/local/zeebe/data
    -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log
    -XX:+ExitOnOutOfMemoryError
    -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000

  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1

  pvcSize: 4Gi
  resources:
    limits:
      memory: 12Gi
      cpu: 8
    requests:
      cpu: 800m

zeebeGateway:
  replicas: 1
  logLevel: debug
  #affinity: {}
  javaOpts: >-
    -XX:+ExitOnOutOfMemoryError
    -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000

  ingress:
    grpc:
      ## @param zeebeGateway.ingress.grpc.enabled if true, an ingress resource is deployed with the Zeebe gateway deployment. Only useful if an ingress controller is available, like nginx.
      enabled: true

      host: "zeebe.local.distro.ultrawombat.com"
      ## @extra zeebeGateway.ingress.grpc.tls configuration for tls on the ingress resource https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
      tls:
        ## @param zeebeGateway.ingress.grpc.tls.enabled if true, then tls is configured on the ingress resource. If enabled the Ingress.host need to be defined.
        enabled: true
        ## @param zeebeGateway.ingress.grpc.tls.secretName defines the secret name which contains the TLS private key and certificate
        secretName: "camunda-platform"

    rest:
      ## @param zeebeGateway.ingress.rest.enabled if true, an ingress resource is deployed with the Zeebe gateway deployment. Only useful if an ingress controller is available, like nginx.
      enabled: true
      host: "zeebe-gateway.local.distro.ultrawombat.com"
      ## @extra zeebeGateway.ingress.rest.tls configuration for tls on the ingress resource https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
      tls:
        ## @param zeebeGateway.ingress.rest.tls.enabled if true, then tls is configured on the ingress resource. If enabled the Ingress.host need to be defined.
        enabled: true
        ## @param zeebeGateway.ingress.rest.tls.secretName defines the secret name which contains the TLS private key and certificate
        secretName: "camunda-platform"

elasticsearch:
  global:
    kibanaEnabled: true
  kibana:
    image:
      repository: bitnami/kibana
      tag: 8.12.2
  master:
    masterOnly: false
    heapSize: 1024m
    persistence:
      size: 16Gi
    replicaCount: 1
  clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"
  replicas: 1

[Ben-Sheppard]

Its been pointed out that setting extraStartupArgs: “--hostname-url=https://keycloak.local.distro.ultrawombat.com/auth” will cause web modeler to fail to start up due to a mismatch in ingress and service url

[jessesimpson36]

related: spring-projects/spring-security#14633

[Ben-Sheppard]

Closing this issue to track at https://github.com/camunda-cloud/identity/issues/2865 due to a change in the scope of the fix.

[jessesimpson36]

related issues
spring-projects/spring-security#14633
keycloak/keycloak#29783