'Terraform 403 error when creating function app and storage account with private endpoint

I am getting a 403 forbidden when creating a function app that connects to its storage account via private endpoint inside a vnet. Storage account has firewall default action of 'Deny', and of course if I set it to 'Allow' it will work. I want this as 'Deny', however. Following this microsoft link if the function app and storage account are created in the same region with vnet, subnets, and private endpoints then it's supposed to work so I must be doing something wrong. I also tried changing the region for the storage account and it still resulted in a 403.

Error:

Error: web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="BadRequest" Message="There was a conflict. The remote server returned an error: (403) Forbidden." Details=[{"Message":"There was a conflict. The remote server returned an error: (403) Forbidden."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"01020","Message":"There was a conflict. The remote server returned an error: (403) Forbidden.","MessageTemplate":"There was a conflict. {0}","Parameters":["The remote server returned an error: (403) Forbidden."]}}]

Here is my terraform code

resource "azurerm_function_app" "func" {
  name                       = "${var.func_basics.name}-func"
  location                   = var.func_basics.location
  resource_group_name        = var.func_basics.resource_group_name
  app_service_plan_id        = azurerm_app_service_plan.svc_plan.id
  storage_account_name       = azurerm_storage_account.func_sa.name
  storage_account_access_key = azurerm_storage_account.func_sa.primary_access_key
  version                    = var.runtime_version
  https_only                 = true
  depends_on = [
    azurerm_storage_account.func_sa,
    azurerm_app_service_plan.svc_plan,
    azurerm_application_insights.func_ai,
    azurerm_virtual_network.func_vnet
  ]

  app_settings = merge(var.app_settings, local.additional_app_settings)

}

resource "azurerm_app_service_plan" "svc_plan" {
  name                = "${var.func_basics.name}-func-plan"
  location            = var.func_basics.location
  resource_group_name = var.func_basics.resource_group_name
  kind                = "elastic"

  sku {
    tier = "ElasticPremium"
    size = "EP1"
  }
}

resource "azurerm_application_insights" "func_ai" {
  name                = "${var.func_basics.name}-func-appi"
  location            = var.func_basics.location
  resource_group_name = var.func_basics.resource_group_name
  application_type    = var.ai_app_type
}

resource "azurerm_storage_account" "func_sa" {
  name                      = "st${lower(replace(var.func_basics.name, "/[-_]*/", ""))}"
  resource_group_name       = var.func_basics.resource_group_name
  location                  = var.func_basics.location
  account_tier              = var.sa_settings.tier
  account_replication_type  = var.sa_settings.replication_type
  account_kind              = "StorageV2"
  enable_https_traffic_only = true
  min_tls_version           = "TLS1_2"
  depends_on = [
    azurerm_virtual_network.func_vnet
  ]

  network_rules {
    default_action = "Deny"
    virtual_network_subnet_ids = [azurerm_subnet.func_endpoint_subnet.id]
    bypass = [
      "Metrics",
      "Logging",
      "AzureServices"
    ]
  }
}

resource "azurerm_virtual_network" "func_vnet" {
  name                = "${var.func_basics.name}-func-vnet"
  resource_group_name = var.func_basics.resource_group_name
  location            = var.func_basics.location
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "func_service_subnet" {
  name                                          = "${var.func_basics.name}-func-svc-snet"
  resource_group_name                           = var.func_basics.resource_group_name
  virtual_network_name                          = azurerm_virtual_network.func_vnet.name
  address_prefixes                              = ["10.0.1.0/24"]
  enforce_private_link_service_network_policies = true

  service_endpoints = ["Microsoft.Storage"]

  delegation {
    name = "${var.func_basics.name}-func-del"

    service_delegation {
      name    = "Microsoft.Web/serverFarms"
      actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
    }
  }
}

resource "azurerm_subnet" "func_endpoint_subnet" {
  name                                           = "${var.func_basics.name}-func-end-snet"
  resource_group_name                            = var.func_basics.resource_group_name
  virtual_network_name                           = azurerm_virtual_network.func_vnet.name
  address_prefixes                               = ["10.0.2.0/24"]
  enforce_private_link_endpoint_network_policies = true
}

resource "azurerm_private_endpoint" "func_req_sa_blob_endpoint" {
  name                = "${var.func_basics.name}-func-req-sa-blob-end"
  resource_group_name = var.func_basics.resource_group_name
  location            = var.func_basics.location
  subnet_id           = azurerm_subnet.func_endpoint_subnet.id

  private_service_connection {
    name                           = "${var.func_basics.name}-func-req-sa-blob-pscon"
    private_connection_resource_id = azurerm_storage_account.func_sa.id
    is_manual_connection           = false
    subresource_names              = ["blob"]
  }
}

resource "azurerm_private_endpoint" "func_req_sa_file_endpoint" {
  name                = "${var.func_basics.name}-func-req-sa-file-end"
  resource_group_name = var.func_basics.resource_group_name
  location            = var.func_basics.location
  subnet_id           = azurerm_subnet.func_endpoint_subnet.id

  private_service_connection {
    name                           = "${var.func_basics.name}-func-req-sa-file-pscon"
    private_connection_resource_id = azurerm_storage_account.func_sa.id
    is_manual_connection           = false
    subresource_names              = ["file"]
  }
}

resource "azurerm_app_service_virtual_network_swift_connection" "func_vnet_swift" {
  app_service_id = azurerm_function_app.func.id
  subnet_id      = azurerm_subnet.func_service_subnet.id
}

locals {
  additional_app_settings = {
    "APPINSIGHTS_INSTRUMENTATIONKEY" = azurerm_application_insights.func_ai.instrumentation_key
    "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING" = azurerm_storage_account.func_sa.primary_connection_string
    "AzureWebJobsStorage"                      = azurerm_storage_account.func_sa.primary_connection_string
    "WEBSITE_VNET_ROUTE_ALL"                   = "1"
    "WEBSITE_CONTENTOVERVNET"                  = "1"
    "WEBSITE_DNS_SERVER"                       = "168.63.129.16"
  }
}
azurenetworkingterraformcloudfirewall


Solution 1:[1]

It seems that it's a common error message when you create an Azure function where the storage account of the function is added to the Virtual Network, read here for more details.

To resolve it, you can use the local-exec Provisioner to invoke the az CLI command to deny the traffic after all of the provisions are finished.

az storage account update --name storage_account_name --resource-group reource_group_name --default-action 'Deny' --bypass 'AzureServices', 'Logging', 'Metrics'

Alternatively, you can separately configure the storage account network rules. You may need to allow your client's IP to access the storage account.

    resource "azurerm_storage_account_network_rules" "test" {
      resource_group_name  = var.resourceGroupName
      storage_account_name = azurerm_storage_account.func_sa.name
    
    
        default_action = "Deny"
       
        bypass = [
          "Metrics",
          "Logging",
          "AzureServices"
        ]
       ip_rules  = ["x.x.x.x"]
       
        depends_on = [
        azurerm_storage_account.func_sa,
        azurerm_app_service_plan.svc_plan,
        azurerm_application_insights.func_ai,
        azurerm_virtual_network.func_vnet,
        azurerm_function_app.func
      ]

}

enter image description here

In addition, there is a possible solution for this similar case on Github.

Solution 2:[2]

I've had this issue in the past and found that it can be resolved as follows. I've tested this on v3.3.0 of the provider using the azurerm_windows_function_app resource. I think currently this is an Azure problem, in that it if you don't supply a share it will try and create one but will be denied. You'd expect this to work if Allow Azure services on the trusted services list to access this storage account is enabled, but webapps aren't trusted.

  • Create your storage account with IP rules and deny
  • Create a share within this for your function app content
  • within the function set the following configuration settings

WEBSITE_CONTENTAZUREFILECONNECTIONSTRING = <storage_account.primary_connection_string>

WEBSITE_CONTENTSHARE = <your share>

WEBSITE_CONTENTOVERVNET = 1

  • In the functions site configuration set the attribute vnet_route_all_enabled = true

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Nancy Xiong
Solution 2 Tim Tharratt

Related Questions