Monitor alerts for AKS failing in the latest AzureRM version

[prabhakarreddy1234]

azurerm_monitor_metric_alert resource provisioning started failing after upgrading to AzureRM: 2.20.0. I Confirm that same configuration works well with AzureRM:2.18.0.

Exception :
Error creating or updating metric alert "kubernetes-pods-failing-alert" (resource group "rg-name"): insights.MetricAlertsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="Alert update failed. Updating from StaticThresholdCriteria and odata.type SingleResourceMultipleMetricCriteria to StaticThresholdCriteria and odata.type MultipleResourceMultipleMetricCriteria is not supported. Activity ID: 2af1fe71-2891-47ec-99b7-6b2b2d53a58b."

I see that there have been some feature enhancements on azurerm_monitor_metric_alert in AzureRM:2.19.0. So just wondering if i need to make any changes to my configuration.

It's happening for all AKS alerts.

• azurerm_2.20.0

resource "azurerm_monitor_metric_alert" "kubernetes-pods-failing-alert" {
  name                = "pod-failure-rate"
  resource_group_name =  azurerm_resource_group.pftrans.name
  scopes              = [azurerm_kubernetes_cluster.pftrans.id]
  description         = "${title(var.environment_name)} - Pods failing in (${azurerm_kubernetes_cluster.pftrans.name})"

  criteria {
    metric_namespace = "Microsoft.ContainerService/managedClusters"
    metric_name      = "kube_pod_status_phase"
    aggregation      = "Average"
    operator         = "GreaterThan"
    threshold        = 1

    dimension {
        name     = "phase"
        operator = "Include"
        values   = ["Failed"]
    }

    dimension {
        name     = "namespace"
        operator = "Include"
        values   = [var.environment_name]
    }
  }

  action {
    action_group_id = data.azurerm_monitor_action_group.xxx.id
  }

  tags = local.common_tags
}

[aristosvo]

Hi @prabhakarreddy1234 ! Thanks for raising this issue. I think I’ve spotted the issue:

As your error message already showed, it is a problem if an already created alert with SingleResourceMultipleMetricCriteria is updated to MultipleResourceMultipleMetricCriteria. After the introduction of the feature in azurerm_monitor_metric_alert all alerts are constructed as MultipleResource, even if there is only one resource.

The easiest solution would probably be to recreate your alerts by tainting them, but I agree that this issue should be solved in the code itself as well.

[aristosvo]

@magodo Do you agree with my temporary solution?

For the definite solution, would you propose tainting based on a catched error message or reimplement it with backwards compatibility?

[magodo]

@prabhakarreddy1234 Apologize for your inconvinience :(

@aristosvo You are right, both the workaround and the root cause analysis. The new AKS cluster will always use MultipleResourceMultipleMetricCriteria`. I'll try figure out a way to make it backwards compatible.

[ghost]

This has been released in version 2.23.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.23.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!