Private AKS cluster, enabled outbound, metrics pod keeps failing

[hardik-id]

Hello,
I have private AKS cluster with open outbound connection and UAMI. I have tried to enable managed prometheus for metrics collections but it seems to fail. Can someone help? I followed this guide https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli

││ prometheus-collector Error: configmap section not mounted, using defaults 
││ addon-token-adapter 2025/02/09 00:21:37 handlers.go:57: received token request, handling...                                                                                                                                                    ││ addon-token-adapter 2025/02/09 00:23:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ addon-token-adapter 2025/02/09 00:24:51 handlers.go:57: received token request, handling...                                                                                                                                                    ││ addon-token-adapter 2025/02/09 00:25:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ addon-token-adapter 2025/02/09 00:27:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ addon-token-adapter 2025/02/09 00:29:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ stream closed EOF for kube-system/ama-metrics-5bff7d784d-mccqf (prometheus-collector)                             
││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 0 minutes                                                                                                                                                                                  ││ prometheus-collector 2025-02-09T00:33:05 No configuration present for the AKS resource                                                                                                                                                         ││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 0 minutes                                                                                                                                                                                  ││ prometheus-collector 2025-02-09T00:33:20 No configuration present for the AKS resource                                                                                                                                                         ││ addon-token-adapter 2025/02/09 00:33:22 utils.go:35: received event type MODIFIED                                                                                                                                                              ││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 1 minutes                                                                                                                                                                                  ││ prometheus-collector TokenConfig.json does not exist                                                                                                                                                                                           ││ prometheus-collector azmon-container-start-time file exists, reading start time                                                                                                                                                                ││ prometheus-collector Container has been running for 1 minutes    
││ prometheus-collector Container has been running for 1 minutes                                                                                                                                                                                  ││ prometheus-collector {"time":1739061266.590147,"filepath":"/opt/microsoft/linuxmonagent/mdsd.err","log":"2025-02-09T00:34:26.5900480Z: [/__w/1/s/external/WindowsAgent/src/shared/mcsmanager/lib/src/RefreshConfigurations.cpp:318,GetAgentCon ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)                                                                                                                                ││ prometheus-collector Metrics Extension is not running (configuration exists)                                                                                                                                                                   ││ prometheus-collector Health check failed: 503, Message: Metrics Extension is not running (configuration exists)   

I tried to look at https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-troubleshoot to find solutions here but it did not help. for example

• Checked if UAMI has metric publish role
• DCE and DCR are created, not sure how they work though