[Issue] Keyvault references in ARM parameter files are not supported

[GABRIELNGBTUC]

• Make sure you've installed the latest version using instructions in the wiki

Output from azd version
Run azd version and copy and paste the output here:

azd version 1.11.0 (commit 5b92e06)

Describe the bug
When deploying the infrastructure with a parameter file making use of keyvault references, azd prompts the user to input a value for the parameter and overrides the value instead of fetching the value from the referenced keyvault + secret name combination.

To Reproduce

1. Use azd init with the minimal solution option
2. Add the following lines to your bicep file

@secure()
param test_kv string 


output test string = test_kv

3. Create a keyvault on Azure and a new secret named "test-kv" with any value inside that keyvault. Also store the id of the keyvault somewhere
4. Modify the parameter file as such

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
      "environmentName": {
        "value": "${AZURE_ENV_NAME}"
      },
      "location": {
        "value": "${AZURE_LOCATION}"
      },
      "test_kv": {
        "reference": {
          "keyVault": {
            "id": "keyvault id your stored at the previous step"
          },
          "secretName": "test-kv"
        }
      }
    }
}

5. Run azd provision
6. You will be prompted for a value for the test_kv parameter. Enter a value different from what is in the keyvault
7. After the deployment, check the deployment output in the portal, the output value test reuses the value provided at step 6 instead of the value of the keyvault set at step 3

Expected behavior
Keyvault references in JSON parameter files are supported and treated the same way as parameters defined with the value property, instead of being hijacked by the AZD cli

Environment
Information on your environment:

• Bicep v0.32
• IDE and version : VS code 1.96.2

Additional context
The error caused when using the --no-prompt switch is also unclear but covered by the issue #3795

[rajeshkamal5050]

@GABRIELNGBTUC thanks for reporting. Can you share format of your keyvault ID, hope its of this format - "id": "/subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.KeyVault/vaults/<vault-name>" ref: Reference secrets with static ID

@JeffreyCA can you triage this? May be AZD is unaware of this key vault referencing in the param file and by default prompting and using it for @secure()?

cc @vhvb1989

[JeffreyCA]

I was able to repro the issue as well. I tried debugging a little and it looks like when azd tries to deserialize the main.parameters.json, it recognizes params in the form:

"parameters": {
  "environmentName": {
    "value": "${AZURE_ENV_NAME}"
  },
  "location": {
    "value": "${AZURE_LOCATION}"
  },
}

but not KeyVault references like:

"parameters": {
  "test_kv": {
    "reference": {
      "keyVault": {
        "id": "/.../"
      },
      "secretName": "test-kv"
    }
  }
}

There might be more going on, but from my initial findings these may be relevant:

azure-dev/cli/azd/pkg/infra/provisioning/bicep/bicep_provider.go

Lines 1524 to 1527 in ad53ef2

	var armParameters azure.ArmParameterFile
	if err := json.Unmarshal([]byte(replaced), &armParameters); err != nil {
	return nil, fmt.Errorf("error unmarshalling Bicep template parameters: %w", err)
	}

azure-dev/cli/azd/pkg/azure/arm_parameters.go

Lines 1 to 20 in ad53ef2

	// Copyright (c) Microsoft Corporation. All rights reserved.
	// Licensed under the MIT License.
	package azure
	// ArmParameters is a map of arm template parameters to their configured values.
	type ArmParameters map[string]ArmParameterValue
	// ArmParametersFile is the model type for a `.parameters.json` file. It fits the schema outlined here:
	// https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json
	type ArmParameterFile struct {
	Schema string `json:"$schema"`
	ContentVersion string `json:"contentVersion"`
	Parameters ArmParameters `json:"parameters"`
	}
	// ArmParameterValue wraps the configured value for the parameter.
	type ArmParameterValue struct {
	Value any `json:"value"`
	}

And a similar issue I found: #1474

[JeffreyCA]

The equivalent of Key Vault references for .bicepparam files (az.getSecret) doesn't seem to be working either:

main.bicepparam (replacing main.parameters.json)

using 'main.bicep'

param environmentName = readEnvironmentVariable('AZURE_ENV_NAME')
param location = readEnvironmentVariable('AZURE_LOCATION')
param test_kv = az.getSecret('<subscription>', '<rg>', '<vault name>', 'test-kv')

azd provision ends up failing with:

ERROR: deployment failed: error deploying infrastructure: starting deployment to subscription: PUT https://management.azure.com/subscriptions/.../providers/Microsoft.Resources/deployments/jcenv-1737140673
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidTemplate
--------------------------------------------------------------------------------
{
  "error": {
    "code": "InvalidTemplate",
    "message": "Deployment template validation failed: 'The value for the template parameter 'test_kv' at line '1' and column '718' is not provided. Please see https://aka.ms/arm-create-parameter-file for usage details.'.",
    "additionalInfo": [
      {
        "type": "TemplateViolation",
        "info": {
          "lineNumber": 1,
          "linePosition": 718,
          "path": "properties.template.parameters.test_kv"
        }
      }
    ]
  }
}

[vhvb1989]

azd does not support this using keyvault references like this, @GABRIELNGBTUC

The main.parameters.json file is used by azd as a template to produce the parameters.json file that is used to submit the ARM deployment. The syntaxis you are using makes ARM server to use the keyvault secret during deployment on the server side, but his is currently blocked/affected by the azd's prompt for parameters feature, as it is reacting to your parameter w/o a default value set.

You can try these workarounds:

A) Add a default value to your parameter in main.bicep like param test_kv string = ""
This will prevent azd from prompting for a value. The parameters.json should still contain your parameter definition and used during deployment to override the default value when the ARM server runs the deployment.

B) You can make azd to pull the value from the KeyVault secret before sending the ARM template. You can do this by using this syntaxis in the main.parameters.json:

"test_kv": {
      "value": "$(secretOrRandomPassword ${AZURE_KEY_VAULT_NAME} test-kv)"
    },

You can either set AZURE_KEY_VAULT_NAME as a key in the .env file (or call azd env set AZURE_KEY_VAULT_NAME fooo), or you can directly type the name of the key vault account name in there. Azd would try to pull the secret test-kv value from that key vault account and use it in the parameters.json that sends to the ARM server. This requires you to have read access to the key vault account secrets. This is a client-side approach. The only problem (side effect) of this approach is that if the key vault account is not found (or the secret), azd generates a random value and tries to save it as a Key Vault secret in the key vault account you provided. This is b/c this feature is mainly for generating passwords that azd can automatically save as key vault secrets.

The FIX we need for azd:

We need to schedule and plan a fix where azd checks if there is a keyvault secret reference definition for a parameter and if yes, skip the prompt.

This might take a few days to have it fixed, so you can try the workarounds if you are blocked. Otherwise you can wait for the fix in a future azd release