How to enable Azure DevOps for Key-vault Access using Terraform

Question

I have an azure devops pipeline with terraform to provision a private keyvault. Since I am using azure managed agents to run the pipeline jobs, it fails as the agent doesn't have access to create/delete/update keyvault certificates. I could have;

1. Created the keyvault with terraform

2. Then stop the pipeline

3. Then manually add agent IP to the kayvault

4. Then run the pipeline for creating rest of the resources (certs, keys etc...)

But in an automatable environment this is not a good practice. What can I do?

I tried local-exec in terraform to get the agent IP and put it in the keyvault with terraform as below. But the problem is, agent doesn't have the access to update the IP in the first place to start with. So, this solution is not a solution

resource "null_resource" "get_agent_ip" {
  triggers = { always_run = "${timestamp()}" }
  provisioner "local-exec" {
    command = "curl https://ipinfo.io/ip >> /tmp/agent_public_ip.txt"
  }
}

data "local_file" "agent-ip" {
  filename   = "/tmp/agent_public_ip.txt"
  depends_on = [null_resource.get_agent_ip]
}

resource "azurerm_key_vault" "cert_kv" {
  name                        = join("-", [var.project, var.environment, var.kv_name])
  location                    = var.location
  resource_group_name         = var.resource_group_name
  enabled_for_disk_encryption = true
  tenant_id                   = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days  = 7
  purge_protection_enabled    = false

  sku_name = var.kv_sku

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
    ip_rules       = [tostring(data.local_file.agent-ip.content)]
    virtual_network_subnet_ids = var.subnet_ids
  }
...

According to this article, the only possible way is to create a self-hosted agent. But I don't want it. Is this possible with service principles or any kind of method?

Answer 1

We use variables when creating Terraform configuration files to be able to change and adapt our code to be reusable. When configuring variables, the best method for me was selecting a single location for both sensitive (secrets) and non-sensitive (resource names, etc..) information, allowing me to manage variables in one place. In this post I will cover how to use variables in Terraform, how to store variables in Azure Key Vault and how to use these variables in Azure DevOps as part of a deployment.

How to configure variables using Terraform

First you need to declare a variable in the Terraform code you are writing. For example:

variable "VMPASS"{
    type = string
}

I would usually put all my variables in a separate file but in the same directory to make it easier for myself to locate and manage them.

Once configured, you can use the variable in the code by replacing the string you would enter with the variable, for example:

password = var.VMPASS

Make sure you commit your code to a repository in Azure DevOps.

Storing variables in Azure Key Vault

Very simple, create your Azure Key Vault if you haven't done so already. From within the Key Vault resource you will need to create a secret by selecting Secrets from the side menu.

Now select Generate/Import.

And here create the name of the secret for the variable and the value you require for your code. Please note Azure Key Vault does not support curtain characters, for example underscore which is something we require when using external sources for variables. Click here for more information but we will cover in the next section how we remap these.

Link Azure DevOps to Key Vault

We now need to link our Azure DevOps to Azure Key Vault. Open your project within Azure DevOps and from the side menu select Pipelines then Library. Here select Variable group.

Give your Variable Group a name and enable the Link secrets from an Azure key vault as variables toggle. From here you want to select the Azure Subscription and Key Vault you created your Terraform variables in (if you haven't linked your Azure subscription to Azure DevOps, use the Manage link to create a Service Principal).

You might be asked to Authorize the access to the Key Vault but once this is done, you can select the Add option to add secrets to your variable library.

Now click Save to complete the library creation.

How to use variables in your pipeline

Finally we need to either edit an existing pipeline or create a new one. We need to include the variable library we created that connects to Key Vault, to do this you need to select the Variables tab and then select Variable groups.

Here select the Link variable group and select the newly created variable group.

Now we need to add a Bash Script task to run remap commands so the Azure Key Vault variables are in the supported format. Terraform expects from an external source the format to be TF_VAR_NAME (underscores not supported in Azure Key Vault and why we have to remap), where name is the variable name. Below is an example of the command we need to add to the bash script task, repeated for each variable that needs to be remapped for Terraform.

echo "##vso[task.setvariable variable=TF_VAR_VMPASS;]$tf-var-vmpass"

Once this is done, you can add the Terraform tasks to install, initialize and plan/deploy. Once the pipeline runs, the script will map the Azure Key Vault variables to new names that can be identified by Terraform.