function_app_resource can't deploy a function app with a backing storage account protected via private endpoint

[cmendible]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.13.5

• provider registry.terraform.io/hashicorp/azurerm v2.50.0
• provider registry.terraform.io/hashicorp/http v2.1.0

Affected Resource(s)

• function_app_resource and possibly app_service_resource

Terraform Configuration Files

Using the configuration files found in this repo will result in and 403 error while trying to deploy the Azure Function: https://github.com/cmendible/azure.samples/tree/function_private_sa_403/function_sa_private_endpoint.v2/deploy

main.tf code:

# Create Resource Group
resource "azurerm_resource_group" "rg" {
  name     = var.resource_group
  location = var.location
}

# Create VNet
resource "azurerm_virtual_network" "vnet" {
  name                = "private-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  # Use Private DNS Zone. That's right we have to add this magical IP here.
  dns_servers = ["168.63.129.16"]
}

# Create the Subnet for the Azure Function. This is thge subnet where we'll enable Vnet Integration.
resource "azurerm_subnet" "service" {
  name                 = "service"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.1.0/24"]

  enforce_private_link_service_network_policies  = true
  enforce_private_link_endpoint_network_policies = true

  # Delegate the subnet to "Microsoft.Web/serverFarms"
  delegation {
    name = "acctestdelegation"

    service_delegation {
      name    = "Microsoft.Web/serverFarms"
      actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
    }
  }
}

# Create the Subnet for the private endpoints. This is where the IP of the private enpoint will live.
resource "azurerm_subnet" "endpoint" {
  name                 = "endpoint"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.2.0/24"]

  enforce_private_link_service_network_policies  = false
  enforce_private_link_endpoint_network_policies = true
}

# Get current public IP. We'll need this so we can access the Storage Account from our PC.
data "http" "current_public_ip" {
  url = "http://ipinfo.io/json"
  request_headers = {
    Accept = "application/json"
  }
}

# Create the "private" Storage Account.
resource "azurerm_storage_account" "sa" {
  name                      = var.sa_name
  resource_group_name       = azurerm_resource_group.rg.name
  location                  = azurerm_resource_group.rg.location
  account_tier              = "Standard"
  account_replication_type  = "GRS"
  enable_https_traffic_only = true
  # We are enabling the firewall only allowing traffic from our PC's public IP.
  network_rules {
    default_action             = var.sa_firewall_enabled ? "Deny" : "Allow"
    bypass                     = ["AzureServices"]
    virtual_network_subnet_ids = []
    ip_rules = [
      jsondecode(data.http.current_public_ip.body).ip
    ]
  }
}

# Create input container
resource "azurerm_storage_container" "input" {
  name                  = "input"
  container_access_type = "private"
  storage_account_name  = azurerm_storage_account.sa.name
}

# Create output container
resource "azurerm_storage_container" "output" {
  name                  = "output"
  container_access_type = "private"
  storage_account_name  = azurerm_storage_account.sa.name
}

# Create the Private endpoint for each Storage Account Service. This is how the Storage account gets the private IPs inside the VNet.
resource "azurerm_private_endpoint" "endpoint" {
  count               = length(var.sa_services)
  name                = "sa-${var.sa_services[count.index]}-endpoint"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  subnet_id           = azurerm_subnet.endpoint.id

  private_service_connection {
    name                           = "sa-${var.sa_services[count.index]}-privateserviceconnection"
    private_connection_resource_id = azurerm_storage_account.sa.id
    is_manual_connection           = false
    subresource_names              = [var.sa_services[count.index]]
  }

  depends_on = [azurerm_storage_share.functions]
}

# Create the blob.core.windows.net Private DNS Zone
resource "azurerm_private_dns_zone" "private" {
  count               = length(var.sa_services)
  name                = "privatelink.${var.sa_services[count.index]}.core.windows.net"
  resource_group_name = azurerm_resource_group.rg.name
}

# Create an A record pointing to each Storage Account service private endpoint
resource "azurerm_private_dns_a_record" "sa" {
  count               = length(var.sa_services)
  name                = var.sa_name
  zone_name           = azurerm_private_dns_zone.private[count.index].name
  resource_group_name = azurerm_resource_group.rg.name
  ttl                 = 3600
  records             = [azurerm_private_endpoint.endpoint[count.index].private_service_connection[0].private_ip_address]
}

# Link the Private Zone with the VNet
resource "azurerm_private_dns_zone_virtual_network_link" "sa" {
  count                 = length(var.sa_services)
  name                  = "networklink-${azurerm_private_dns_zone.private[count.index].name}"
  resource_group_name   = azurerm_resource_group.rg.name
  private_dns_zone_name = azurerm_private_dns_zone.private[count.index].name
  virtual_network_id    = azurerm_virtual_network.vnet.id
  registration_enabled  = false
}

resource "azurerm_storage_share" "functions" {
  name                 = "${var.func_name}-content"
  storage_account_name = azurerm_storage_account.sa.name
}

# Create the Azure Function plan (Elastic Premium) 
resource "azurerm_app_service_plan" "plan" {
  name                = "azure-functions-test-service-plan"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  kind = "elastic"
  sku {
    tier     = "ElasticPremium"
    size     = "EP1"
    capacity = 1
  }
  maximum_elastic_worker_count = 20
}

# Create Application Insights
resource "azurerm_application_insights" "ai" {
  name                = var.func_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  application_type    = "web"
}

# Create the Azure Function App
resource "azurerm_function_app" "func_app" {
  name                       = var.func_name
  location                   = azurerm_resource_group.rg.location
  resource_group_name        = azurerm_resource_group.rg.name
  app_service_plan_id        = azurerm_app_service_plan.plan.id
  storage_account_name       = azurerm_storage_account.sa.name
  storage_account_access_key = azurerm_storage_account.sa.primary_access_key
  version                    = "~3"
  https_only                 = true

  app_settings = {
    APPINSIGHTS_INSTRUMENTATIONKEY        = azurerm_application_insights.ai.instrumentation_key
    APPLICATIONINSIGHTS_CONNECTION_STRING = "InstrumentationKey='${azurerm_application_insights.ai.instrumentation_key}'"
    FUNCTIONS_WORKER_RUNTIME = "dotnet"
    WEBSITE_VNET_ROUTE_ALL  = "1"
    WEBSITE_CONTENTOVERVNET = "1"
    WEBSITE_DNS_SERVER      = "168.63.129.16"
  }

  depends_on = [
    azurerm_storage_account.sa,
    azurerm_private_endpoint.endpoint,
    azurerm_private_dns_a_record.sa,
    azurerm_private_dns_zone_virtual_network_link.sa
  ]
}

# Enable Regional VNet integration. Function --> service Subnet 
resource "azurerm_app_service_virtual_network_swift_connection" "vnet_integration" {
  app_service_id = azurerm_function_app.func_app.id
  subnet_id      = azurerm_subnet.service.id
}

Expected Behaviour

The Azure Function App should be deployed without issues when the backing storage account is protected via private endpoint

Actual Behaviour

The Azure Function App deployment fails with a 403 exception.

The problem is caused by the way app_settings are used by the provider. To be able to deploy an Azure Function connected to a backing storage account protected via private endpoint the following app_settings must be set when the app is created:

    WEBSITE_VNET_ROUTE_ALL  = "1"
    WEBSITE_CONTENTOVERVNET = "1"
    WEBSITE_DNS_SERVER      = "168.63.129.16"

Current provider implementation only sets storage account (basic) related app_settings when creating the app:

https://github.com/terraform-providers/terraform-provider-azurerm/blob/bb82a8e3c343add6abb011256779a619811cb954/azurerm/internal/services/web/function_app_resource.go#L298

That behavior blocks the correct deployment of the Function App. To prove it I created a fork and implemented a quick & dirty workaround : cmendible@bceefa4 expanding all app_settings before the function app is created. The resulting provider deployed the function app as expected.

The question is why aren't the app_settings respected when the function is first deployed, but updated after creation? What would be the best way to fix this? Should we create variables for the 3 parameters shown above and make them part of the basic settings?

Steps to Reproduce

1. terraform apply

Important Factoids

References

[phatcher]

Had this recently and it's an Azure rather than terraform issue. It does mean you need a multi-stage approach to deployment though...

1. Unsecure the storage account
2. Deploy the function app, not on the vnet
3. Put the function app on the vnet
4. Resecure the storage account

The reasoning is that when you deploy the function app telling it is on the vnet, it is not actually on the vnet yet - this confuses the Azure backplane

[cmendible]

Hi @phatcher that used to be the case, and I even wrote about it in the update to my post here: Azure Functions: use Blob Trigger with Private Endpoint. But since the introduction of the WEBSITE_CONTENTOVERVNET parameter there is no longer a need to deploy the Function App using the steps you describe.

You can try the following Azure ARM template, created by @gabesmsft, to deploy a Function App, with VNET integration using a backing storage account protected via private endpoint, without issues https://github.com/cmendible/FunctionAppWithStorageEndpointsARM or you can build my version of the provider with the "dirty" fix (cmendible@bceefa4) and apply the provided terraform configuration and it'll deploy a Function App, as described, in one go and without issues.

[phatcher]

@cmendible Thanks for that, explains the behaviour I had - I spent a number of hours recently with MS support trying to get it to work.

For Linux docker apps I found you also need the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE, otherwise the function app failed to start, seems to use this location for the Kudu environment/logs.

The changes should also be made on the slot resources as well.

[cmendible]

Yep those values are set here:

https://github.com/terraform-providers/terraform-provider-azurerm/blob/bb82a8e3c343add6abb011256779a619811cb954/azurerm/internal/services/web/function_app.go#L217

by the provider when you set the following properties in the terraform configuration:

storage_account_name       = azurerm_storage_account.sa.name
storage_account_access_key = azurerm_storage_account.sa.primary_access_key

also the value of WEBSITE_CONTENTSHARE is always set to <name of the function app>-content:

https://github.com/terraform-providers/terraform-provider-azurerm/blob/bb82a8e3c343add6abb011256779a619811cb954/azurerm/internal/services/web/function_app.go#L249

[phatcher]

The portal/arm template set WEBSITE_CONTENTSHARE it to <name of function-app><random 4 char hex> which is what I've replicated in my module. I think this avoids the potential of collisions when multiple apps are deployed over time. ISTR there was an issue on my project where we are using ElasticPremium and durable functions with failover and also needing slot swap deployment, where we needed to split the management account (and keep it geo-local) from the task hub location, see durable-functions zero downtime deployment

I was also wondering whether the provider should support RBAC storage account access OOB rather than using the access key i.e. have it grant the Storage Blob Data Contributor etc rights if Managed Identity is set on the function app, but this is probably a module rather than resource provider level decision.

[scuderig]

I'm experiencing an issue deploying an azure function on ASE v3, and it seems again related to the way the provider set the parameters on the app as explained above.

This is the error which is shown by the control plan of Azure when trying to deploy:

The parameter 'appsettings' has an invalid value. Details: On ASEv3, the app setting, WEBSITE_VNET_ROUTE_ALL, must exist and its value must be 1.

Useless to say WEBSITE_VNET_ROUTE_ALL is set to 1 in my terraform code

Overall experience deploying Azure functions is terrible, someone really needs to look into it please

[f0o]

@cmendible

Yep those values are set here:

https://github.com/terraform-providers/terraform-provider-azurerm/blob/bb82a8e3c343add6abb011256779a619811cb954/azurerm/internal/services/web/function_app.go#L217

by the provider when you set the following properties in the terraform configuration:

storage_account_name       = azurerm_storage_account.sa.name
storage_account_access_key = azurerm_storage_account.sa.primary_access_key

also the value of WEBSITE_CONTENTSHARE is always set to <name of the function app>-content:

https://github.com/terraform-providers/terraform-provider-azurerm/blob/bb82a8e3c343add6abb011256779a619811cb954/azurerm/internal/services/web/function_app.go#L249

It would be great if you could override those... I've been chasing ghosts until I noticed that TF is actually overriding my values there.

Is this worth an own issue?

[ZachTB123]

The same issue exists with the azurerm_logic_app_standard resource where it is only setting the basic app settings on initial creation.

[cmendible]

@cmendible

Yep those values are set here:

https://github.com/terraform-providers/terraform-provider-azurerm/blob/bb82a8e3c343add6abb011256779a619811cb954/azurerm/internal/services/web/function_app.go#L217

by the provider when you set the following properties in the terraform configuration:

storage_account_name       = azurerm_storage_account.sa.name
storage_account_access_key = azurerm_storage_account.sa.primary_access_key

also the value of WEBSITE_CONTENTSHARE is always set to <name of the function app>-content:

https://github.com/terraform-providers/terraform-provider-azurerm/blob/bb82a8e3c343add6abb011256779a619811cb954/azurerm/internal/services/web/function_app.go#L249

It would be great if you could override those... I've been chasing ghosts until I noticed that TF is actually overriding my values there.

Is this worth an own issue?

The PR I submitted ahould also help with this one.

I'll try to add another ASAP.

[cmendible]

The same issue exists with the azurerm_logic_app_standard resource where it is only setting the basic app settings on initial creation.

Did you create an issue for this one?

[f0o]

The same issue exists with the azurerm_logic_app_standard resource where it is only setting the basic app settings on initial creation.

Did you create an issue for this one?

I did not hence the question before I create some inflationary issue :)

[danielrobinson95]

@cmendible Any idea when this fix will make it to the next version of the azurerm provider?

[cmendible]

waiting for @jackofallops's review on PR #14521

[arpitcpanchal]

Hi @cmendible ,
Thanks for the insight here.
I was recently looking around the deploying a function app (linux, EP1) with vnet integration and backing storage account with private endpoint for both blob and file. With intent of traffic from function app to storage account goes via vnet to relevant private endpoint (blob/file).
However, was facing similar issues. The terraform deployment seemed to fail with current provider version v2.89.0, where it complained about storage file share creation permission.

Having further look, it seems the 2.77.0 had some updates around WEBSITE_CONTENTSHARE, WEBSITE_CONTENTAZUREFILECONNECTIONSTRING in the app settings.
I tried deploying function app with vnet integration and backing storage with Private endpoint using the v2.76.0, which seemed to work fine.

Wondering if the error in v2.89.0 around creation of file share are because of the changes introduced in 2.77.0 or something with the rbac on the azure files itself?

[cmendible]

Hi @cmendible , Thanks for the insight here. I was recently looking around the deploying a function app (linux, EP1) with vnet integration and backing storage account with private endpoint for both blob and file. With intent of traffic from function app to storage account goes via vnet to relevant private endpoint (blob/file). However, was facing similar issues. The terraform deployment seemed to fail with current provider version v2.89.0, where it complained about storage file share creation permission.

Having further look, it seems the 2.77.0 had some updates around WEBSITE_CONTENTSHARE, WEBSITE_CONTENTAZUREFILECONNECTIONSTRING in the app settings. I tried deploying function app with vnet integration and backing storage with Private endpoint using the v2.76.0, which seemed to work fine.

Wondering if the error in v2.89.0 around creation of file share are because of the changes introduced in 2.77.0 or something with the rbac on the azure files itself?

Issue was not related to RBAC but with how the provider configures the Function App Settings. Values such as:

WEBSITE_VNET_ROUTE_ALL  = "1"
WEBSITE_CONTENTOVERVNET = "1"  

must be respected at creation time and that is being addressed here: #14638

[arpitcpanchal]

Hi @cmendible , Thanks for the insight here. I was recently looking around the deploying a function app (linux, EP1) with vnet integration and backing storage account with private endpoint for both blob and file. With intent of traffic from function app to storage account goes via vnet to relevant private endpoint (blob/file). However, was facing similar issues. The terraform deployment seemed to fail with current provider version v2.89.0, where it complained about storage file share creation permission.
Having further look, it seems the 2.77.0 had some updates around WEBSITE_CONTENTSHARE, WEBSITE_CONTENTAZUREFILECONNECTIONSTRING in the app settings. I tried deploying function app with vnet integration and backing storage with Private endpoint using the v2.76.0, which seemed to work fine.
Wondering if the error in v2.89.0 around creation of file share are because of the changes introduced in 2.77.0 or something with the rbac on the azure files itself?

Issue was not related to RBAC but with how the provider configures the Function App Settings. Values such as:

WEBSITE_VNET_ROUTE_ALL  = "1"
WEBSITE_CONTENTOVERVNET = "1"  

must be respected at creation time and that is being addressed here: #14638

ah okay. Thanks @cmendible

[github-actions]

This functionality has been released in v2.90.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[danielrobinson95]

@cmendible Looks like the CONTENTSHARE value in the function app is no longer the function name appended with "-content". The latest version is appending a random GUID. Do you know how we can assign this value at the time the function app is provisioned?

[jackofallops]

@cmendible Looks like the CONTENTSHARE value in the function app is no longer the function name appended with "-content". The latest version is appending a random GUID. Do you know how we can assign this value at the time the function app is provisioned?

Hi @danielrobinson95 - the use of -content as a fixed value caused problems when using slots, so a unique value was added to allow correct operation there (the Portal takes this same approach). The provider will only do this for new resources, and will continue to honour -content for existing resources.

If you need to specifically use the -content suffix on a new resource for some reason, I think you should be fine to explicitly configure WEBSITE_CONTENTSHARE in app_settings, but you'll also need to explicitly specify WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and the correct value for the storage account you're using.

[danielrobinson95]

@cmendible Looks like the CONTENTSHARE value in the function app is no longer the function name appended with "-content". The latest version is appending a random GUID. Do you know how we can assign this value at the time the function app is provisioned?

Hi @danielrobinson95 - the use of -content as a fixed value caused problems when using slots, so a unique value was added to allow correct operation there (the Portal takes this same approach). The provider will only do this for new resources, and will continue to honour -content for existing resources.

If you need to specifically use the -content suffix on a new resource for some reason, I think you should be fine to explicitly configure WEBSITE_CONTENTSHARE in app_settings, but you'll also need to explicitly specify WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and the correct value for the storage account you're using.

Thanks for the reply @jackofallops. Unfortunately, this does not work. I've explicitly defined both WEBSITE_CONTENTSHARE and WEBSITE_CONTENTAZUREFILECONNECTIONSTRING in my app_settings block, but the value is still overwritten by the random GUID.

[danielrobinson95]

@jackofallops I just retested on v2.91.0 and this is still not working.

[cmendible]

@jackofallops just tests provider v2.93.1 and this is not working.

Setting WEBSITE_CONTENTSHARE is a must in order to deploy Azure Functions with a Private Endpoint protected Storage Account.

I'll continue discussion here: #14167

[nela]

WEBSITE_CONTENTSHARE seems only aupported by Windows workloads. How to get around this when using Linux workloads?

https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_contentshare
https://docs.microsoft.com/en-us/azure/azure-functions/functions-infrastructure-as-code#linux

[cmendible]

WEBSITE_CONTENTSHARE seems only aupported by Windows workloads. How to get around this when using Linux workloads?

https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_contentshare
https://docs.microsoft.com/en-us/azure/azure-functions/functions-infrastructure-as-code#linux

Works for both Windows & Linux. You can try the ARM / bicep configuration provided here: https://github.com/Azure/azure-quickstart-templates/blob/master/quickstarts/microsoft.web/function-app-storage-private-endpoints/main.bicep

[scuderig]

This still does not work for me, either using private endpoint or service endpoint restricting a specific VNet.
I'm not able to upgrade the provider for an existing setup which is perfectly working with a previous version.

It was working with AzureRM provider version 2.67.0
It stopped working with 2.73.0 and above (I tried with 2.97.0 and it still does not work).

The functions are deployed with a linux App Service Plan:
kind = "elastic"
tier = ElasticPremium
size = EP2

The functions are working fine, and deploy was successful with 2.67.0 (bith first deploy and subsequent updates).

Could someone please look at this?

[dawsonar802]

Following for resolution, however I wanted to post some notes on what we saw as well.

• Version 2.89 of the azurerm provider seemed to seemed to deploy the azurerm_function_app resource ok, however we had to add a toggle to deploy initially with the private endpoint off, then deploy again to enable it, otherwise no file share was created and a 403 was thrown.
• Due some needs of another module we updated to version 2.96 of the provider, and while the above issue was not seen, the file share was not created despite the mentioned app settings WEBSITE_CONTENTSHARE and WEBSITE_CONTENTAZUREFILECONNECTIONSTRING. This cause 503 errors in the pipelines to deploy the functions and also
• Kudu/SCM was giving 503 Service Unavailable.
• We also tried 2.98 of the provider but found it to have the same issues.

Is it recommended to move to the OS specific versions of the the function_app resources or do those have similar issues?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.