Unable to bind trusted_root_certificate with backend_http_settings for "azurerm_application_gateway"

[wgoulet]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.9

• provider.azurerm v1.34.0

Affected Resource(s)

• "azurerm_application_gateway"

Terraform Configuration Files

variable "client_secret" {}

variable "certpass" {
  type = string
}

provider "azurerm" {
  # Whilst version is optional, we /strongly recommend/ using it to pin the version of the Provider being used
  version = "=1.34.0"

  subscription_id = "XXXXXXXXXXXX"
  client_id       = "XXXXXXXXXXXX"
  client_secret   = "${var.client_secret}"
  tenant_id       = "XXXXXXXXXXXX"
}


resource "azurerm_application_gateway" "demoapps" {
    disabled_ssl_protocols = []
    enable_http2           = false
    location               = "eastus"
    name                   = "demoappslb"
    resource_group_name    = "vcloud4app"
    tags                   = {}
    zones                  = []

    sku {
        name = "Standard_v2"
        tier = "Standard_v2"
    }

    autoscale_configuration {
        max_capacity = 10
        min_capacity = 2
    }

    backend_address_pool {
        fqdns           = [
            "e2ev1.thehotelcook.com"
        ]
        name            = "demoappsbackend"
    }

    backend_http_settings {
        cookie_based_affinity               = "Disabled"
        name                                = "demoapplbbe"
        pick_host_name_from_backend_address = true
        port                                = 443
        probe_name                          = "demoapplbbef353c2a6-2713-42fe-a369-0d4b8d67500f"
        protocol                            = "Https"
        request_timeout                     = 20
    }

    trusted_root_certificate {
          name = "beroot"
          data = file("${path.module}/cacerts.pem")
    }

    frontend_ip_configuration {
        name                          = "appGwPublicFrontendIp"
        private_ip_address_allocation = "Dynamic"
        public_ip_address_id = "/subscriptions/XXXXXXXXXXXX-XXXXXXXXXXXX-XXXXXXXXXXXX-XXXXXXXXXXXX-XXXXXXXXXXXX/resourceGroups/vcloud4app/providers/Microsoft.Network/publicIPAddresses/demoapppip"
    }

    frontend_port {
        name = "port_443"
        port = 443
    }

    gateway_ip_configuration {
        name      = "appGatewayIpConfig"
        subnet_id = "/subscriptions/XXXXXXXXXXXX-XXXXXXXXXXXX-XXXXXXXXXXXX-XXXXXXXXXXXX-XXXXXXXXXXXX/resourceGroups/vcloud4app/providers/Microsoft.Network/virtualNetworks/demolbvnet/subnets/default"
    }

    http_listener {
        frontend_ip_configuration_name = "appGwPublicFrontendIp"
        frontend_port_name             = "port_443"
        name                           = "demoapplblistener"
        protocol                       = "Https"
        ssl_certificate_name           = "ssl_pubcert"
        require_sni                    = false
    }

    ssl_certificate {
        name = "ssl_pubcert"
        password = "${var.certpass}"
        data = file("${path.module}/tcertp12.b64")
    }
    probe {
        interval                                  = 30
        minimum_servers                           = 0
        name                                      = "demoapplbbef353c2a6-2713-42fe-a369-0d4b8d67500f"
        path                                      = "/"
        pick_host_name_from_backend_http_settings = true
        protocol                                  = "Https"
        timeout                                   = 30
 unhealthy_threshold                       = 3

        match {
            status_code = []
        }
    }

    request_routing_rule {
        backend_address_pool_name  = "demoappsbackend"
        backend_http_settings_name = "demoapplbbe"
        http_listener_name         = "demoapplblistener"
        name                       = "routehttptohttps"
        rule_type                  = "Basic"
    }

}

Debug Output

Panic Output

Expected Behavior

I expected that the trusted root certificate I had uploaded would be associated with backend_http_settings as a trusted certificate so that the StandardV2 Azure App Gateway would properly establish HTTPS connections with my backend that are using certs issued by my private CA.

Actual Behavior

No errors were returned, but when I look at the Azure App Gateway export template for the backend http settings that were created I can clearly see that the root certificate is not associated with the settings.

                "backendHttpSettingsCollection": [
                    {
                        "name": "demoapplbbe",
                        "properties": {
                            "port": 443,
                            "protocol": "Https",
                            "cookieBasedAffinity": "Disabled",
                            "pickHostNameFromBackendAddress": true,
                            "affinityCookieName": "ApplicationGatewayAffinity",
                            "requestTimeout": 20,
                            "probe": {
                                "id": "[concat(resourceId('Microsoft.Network/applicationGateways', parameters('applicationGateways_demoappslb_name')), '/probes/demoapplbbef353c2a6-2713-42fe-a369-0d4b8d67500f')]"
                            }
                        }
                    },

Note that when I used the Azure portal to manually create a backend I can associate the root certificate that I created with trusted_root_certificate with the backend http settings as shown here:

{
                        "name": "test",
                        "properties": {
                            "port": 443,
                            "protocol": "Https",
                            "cookieBasedAffinity": "Disabled",
                            "pickHostNameFromBackendAddress": true,
                            "requestTimeout": 20,
                            "trustedRootCertificates": [
                                {
                                    "id": "[concat(resourceId('Microsoft.Network/applicationGateways', parameters('applicationGateways_demoappslb_name')), '/trustedRootCertificates/beroot')]"
                                }
                            ]
                        }
                    }
                ],

Steps to Reproduce

1. terraform apply

Important Factoids

None

References

It appears the trusted_root_certificate property was added via this request #4206 to specifically address the exact problem I'm trying to solve reported here #3115 but it seems like the solution is not quite complete unless we can specifically associate the uploaded root with the backend_http_settings.

• #0000

[egorchabala]

I think there are two problems:

1. Trusted root certificate can't be attached to backend using current version of provider
2. Looks like Azure doesn't support PEM certificates. I was able to manually attach certificate when switched to CER in TF configuration

...
trusted_root_certificate {
        name = "name"
        data = filebase64("data/certificate.cer")
    }
...

[wgoulet]

@egorchabala I didn't have an issue uploading the certificate when it was encoded in PEM format. But I agree I have not found a solution for 1 and it's good to get independent verification from something else.

I think it is possible to workaround this with a provisioner where we wrap up the Azure RM CLI commands to add the root certificate name to the backend http settings.

[LaurentLesle]

For 1 have you tried to combine the root, any intermediate cert and the cert in the same file?

[wgoulet]

@LaurentLesle I don't think that solves problem 1 because we are specifically trying to solve the problem of ensuring that the trusted_root_certificate is honored by the backend_http_settings. The frontend SSL cert already contains the full cert chain because you are uploading a PKCS12 file that has the full chain + cert & key.

@egorchabala tagging you in case you would benefit from this workaround.

The workaround I have come up with until the provider supports this scenarios is to add two null resources to my terraform plan:

resource "null_resource" "az_login" {
    depends_on = ["azurerm_application_gateway.examplelb "]
    triggers = {
        timestamp = timestamp()
    }
    provisioner "local-exec" {
        command = "az login --service-principal -u XXXXXX -p ${var.client_secret} --tenant  XXXXX"
    }
}

resource "null_resource" "bind_root_http_settings" {
    depends_on = ["null_resource.az_login"]
    triggers = {
        timestamp = timestamp()
    }
    provisioner "local-exec" {
        command = "az network application-gateway http-settings update --gateway-name examplelb --resource-group example-gp --root-cert beroot --name demoapplbbe"
    }
}

I used the current time as a trigger to ensure that the null resources are always executed when I apply changes; that's probably not needed for most scenarios but for testing out the solution it worked well as I could verify the changes applied correctly by deleting the binding in the Azure portal and re-running my plan.

[runecalico]

Even with the worth around, there are a few points that make this less then ideal for an "always on" application gateway.

1. Occasionally the setting on the application gateway will "revert" back to using the well known CA cert. I was unable to see any information in the plan on it, so it might happen silently
2. The workaround is appreciated, but not ideal when you have a number of http settings to update, which happen after the deploy, effectively taking down the application gateway until the private CA certs is re-applied to the http settings. And can't run concurrently (so set parallelism to 1 on apply)

[wgoulet]

@runecalico Definitely agree this workaround has multiple disadvantages; shelling out to OpenSSL is definitely not an ideal solution. For point 2, I'm not sure; it seems the provider should be able to handle it. I'm encouraged that this was marked as a bug so hopefully there will be a solution soon!

[majimenez-stratio]

I think I came up with a solution for this, see the PR above.

[wgoulet]

Thanks for letting me know; following the PR.

[favoretti]

Since this issue seems to have been addressed in the latest versions of the provider - I'm going to close it. Please open a new updated bug report if this is still relevant. Thank you.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.