Support for Azure AKS

[lborupj]

Hi,

I've created a small patch for 0.27.0 which is based on an idea I had to get Headlamp for work with Azure OIDC and AKS clusters.
The idea is to still extract the id-token but also extract the access-token (after doing code exchange) and adding the access-token to the cache with the id-token as the key (as is the case for the refresh-token).

Then, whenever at request is received in the backend, the middleware function will lookup the access-token and replace the Authorization header with this token, before sending it to the Kubernetes API - AKS does not support using id-tokens as access-tokens.
This also works for WSS requests where, obviously, I need to extract the id-token from the wssSecProtocols header rather than the Authorization header.

and it works for my purpose, which is always AKS clusters, but the question is, how to make this more generic and would it be something someone who is more proficient in Headlamp development, be interested in doing this?

(why put it in cache, rather than just returning it to frontend instead of the id-token? Access token tend to be rather large and I ran into issues with browsers not sending the request, and various ingress/L7-LBs not being able to handle 20k+ HTTP headers, so it is better to keep this on the backend). By still sending id-token to frontend, you could extract info if you want to show username and id info stuff in the UI also.)

[illume]

Sounds nice. Do you want to share your preliminary patch?

Maybe adding a cmd line flag/config could enable this feature.

Did you come across any relevant docs in your research I could read?

[lborupj]

sure - still prelim/POC and there might still some debug logs but does work when running in-cluster in AKS. Patchable on 0.27.0

diff --git a/backend/cmd/headlamp.go b/backend/cmd/headlamp.go
index 87cb557c..f83180c0 100644
--- a/backend/cmd/headlamp.go
+++ b/backend/cmd/headlamp.go
@@ -619,13 +619,17 @@ func createHeadlampHandler(config *HeadlampConfig) http.Handler {
                                return
                        }
 
-                       if err := config.cache.Set(context.Background(),
-                               fmt.Sprintf("oidc-token-%s", rawIDToken), oauth2Token.RefreshToken); err != nil {
+                       if err := config.cache.Set(context.Background(), fmt.Sprintf("oidc-token-%s", rawIDToken), oauth2Token.RefreshToken); err != nil {
                                logger.Log(logger.LevelError, nil, err, "failed to cache refresh token")
                                http.Error(w, "Failed to cache refresh token: "+err.Error(), http.StatusInternalServerError)
-
                                return
                        }
+                       if err := config.cache.Set(context.Background(), fmt.Sprintf("oidc-access-token-%s", rawIDToken), oauth2Token.AccessToken); err != nil {
+                               logger.Log(logger.LevelError, nil, err, "failed to cache access-token")
+                               http.Error(w, "Failed to cache access-token: " + err.Error(), http.StatusInternalServerError)
+                               return
+                       }
+
                        idToken, err := oauthConfig.Verifier.Verify(oauthConfig.Ctx, rawIDToken)
                        if err != nil {
                                logger.Log(logger.LevelError, nil, err, "failed to verify ID Token")
@@ -809,12 +813,15 @@ func refreshAndCacheNewToken(oidcAuthConfig *kubeconfig.OidcConfig,
                return "", err
        }
 
-       idToken, ok := tk.Extra("id_token").(string)
+               idToken, ok := tk.Extra("id_token").(string)
        if ok {
                // update cache
                if err := cache.Set(context.Background(), fmt.Sprintf("oidc-token-%s", idToken), tk.RefreshToken); err != nil {
                        logger.Log(logger.LevelError, nil, err, "failed to cache refresh token")
-
+                       return "", err
+               }
+               if err := cache.Set(context.Background(), fmt.Sprintf("oidc-access-token-%s", idToken), tk.AccessToken); err != nil {
+                       logger.Log(logger.LevelError, nil, err, "failed to cache access token")
                        return "", err
                }
 
@@ -842,10 +849,31 @@ func (c *HeadlampConfig) OIDCTokenRefreshMiddleware(next http.Handler) http.Hand
 
                // parse cluster and token
                cluster, token := parseClusterAndToken(r)
-               if cluster == "" || token == "" {
+               if cluster == "" {
                        next.ServeHTTP(w, r)
                        return
                }
+               if token == "" {
+                       wssSecProtocolHeader := r.Header.Get("sec-websocket-protocol")
+                       if wssSecProtocolHeader == "" {
+                               next.ServeHTTP(w, r)
+                               return
+                       }
+                       wssSecProtocols := strings.Split(wssSecProtocolHeader, ",")
+                       for _, e := range wssSecProtocols {
+                               trimmed := strings.TrimSpace(e)
+                               if strings.HasPrefix(trimmed, "base64url.bearer.authorization.k8s.io.") {
+                                       base64EncodedKey := strings.TrimPrefix(trimmed, "base64url.bearer.authorization.k8s.io.")
+                                       tempToken, err := base64.StdEncoding.DecodeString(base64EncodedKey)
+                                       if err != nil {
+                                               logger.Log(logger.LevelError, map[string]string{"cluster": cluster}, err, "failed to decode bearer token")
+                                               next.ServeHTTP(w, r)
+                                               return
+                                       }
+                                       token = string(tempToken)
+                               }
+                       }
+               }
 
                // get oidc config
                kContext, err := c.kubeConfigStore.GetContext(cluster)
@@ -853,7 +881,6 @@ func (c *HeadlampConfig) OIDCTokenRefreshMiddleware(next http.Handler) http.Hand
                        logger.Log(logger.LevelError, map[string]string{"cluster": cluster},
                                err, "failed to get context")
                        next.ServeHTTP(w, r)
-
                        return
                }
 
@@ -864,8 +891,17 @@ func (c *HeadlampConfig) OIDCTokenRefreshMiddleware(next http.Handler) http.Hand
                        return
                }
 
+               // Translate id-token to access-token
+               accessToken, err := c.cache.Get(context.Background(), fmt.Sprintf("oidc-access-token-%s", token));
+               if (err != nil) {
+                       logger.Log(logger.LevelInfo, map[string]string{"cluster": cluster}, err, "failed to get access-token, using id-token");
+               } else {
+                       token = accessToken.(string)
+               }
+
                // skip if token is not about to expire
                if !isTokenAboutToExpire(token) {
+                       r.Header.Set("Authorization", fmt.Sprintf("Bearer %s", token))
                        next.ServeHTTP(w, r)
                        return
                }
diff --git a/backend/cmd/multiplexer.go b/backend/cmd/multiplexer.go
index 06cefc31..8f083ceb 100644
--- a/backend/cmd/multiplexer.go
+++ b/backend/cmd/multiplexer.go
@@ -409,6 +409,7 @@ func (m *Multiplexer) reconnect(conn *Connection) (*Connection, error) {
 
 // HandleClientWebSocket handles incoming WebSocket connections from clients.
 func (m *Multiplexer) HandleClientWebSocket(w http.ResponseWriter, r *http.Request) {
+       fmt.Println(fmt.Sprintf("Got WSS request %s from %s", r.RequestURI, r.RemoteAddr))
        clientConn, err := m.upgrader.Upgrade(w, r, nil)
        if err != nil {
                logger.Log(logger.LevelError, nil, err, "upgrading connection")

[illume]

I did a bit of research, and think access tokens are indeed not really standardized with different providers and K8s. It's not clear to me yet if there are other ones which could reuse this way of doing things. But I expect this could be an AKS only way.

Probably we could abstract out token handling for different providers? If we separate it out the logic will probably be simpler to follow, even if it has some redundancy. I expect we can detect that it's AKS, and use these access tokens. With detection of AKS then we probably wouldn't need to use config/command line flags?

btw. I asked some other Headlamp developers to have a look at this discussion. @yolossn what do you think of the providers idea?

[yolossn]

@illume this looks like a problem specific to Azure OIDC because the returned tokens are big and possibly can happen with other OIDC providers. Though this solution does make sense for this case it can affect other OIDC providers,I don't think all OIDC providers return both Access and ID token when we do a token exchange.

We should explore what is causing the token to be big and check if limiting the scopes can help reduce the size of the access token or we should see if there is a way to remove the unwanted claims and other information from the token.

[lborupj]

We should explore what is causing the token to be big and check if limiting the scopes can help reduce the size of the access token or we should see if there is a way to remove the unwanted claims and other information from the token.

Well, I've looked long and hard for a way to limit claims and especially number of roles returned in Azure but found no solution, and you can't change the token once it's been issued.
I don't actually know why Headlamp uses the id-token for the backend resource (Kubernetes) and not the access-token which is always returned in OIDC auth code flows and also OAuth2. The access token is meant to grant access to an API, in this case Kubernetes API and why Azure will create an access token with an audience (aud) claim for the generic AKS service.
By securing the token in the backend rather than sending it to the frontend (kinda BFF, see https://curity.io/blog/token-handler-the-single-page-applications-new-bff/ ) we also avoid sending tokens between SPA and backend, since all requests are proxied by backend anyways.

I don't know exactly how the current OIDC option is used, but if we can settle on a design, I'd be interested in helping (with my limited Go skills :) )