Nextflow with Azure Batch - Cannot find a matching VM image

Question

While trying to set up Nextflow with Azure Batch (NF-Core), I am getting following error. I tried this on multiple workflows (sarek, ataseq etc.) I get the same error -

N E X T F L O W  ~  version 22.04.0
Pulling nf-core/atacseq ...
 downloaded from https://github.com/nf-core/atacseq.git
Launching `https://github.com/nf-core/atacseq` [rhl6d5529] DSL1 - revision: 1b3a832db5 [1.2.1]
Downloading plugin [email protected]
----------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/atacseq v1.2.1
----------------------------------------------------

Run Name             : rhl6d5529
Data Type            : Paired-End
Design File          : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/design.csv
Genome               : Not supplied
Fasta File           : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/reference/genome.fa
GTF File             : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/reference/genes.gtf
Mitochondrial Contig : MT
MACS2 Genome Size    : 1.2E+7
Min Consensus Reps   : 1
MACS2 Narrow Peaks   : No
MACS2 Broad Cutoff   : 0.1
Trim R1              : 0 bp
Trim R2              : 0 bp
Trim 3' R1           : 0 bp
Trim 3' R2           : 0 bp
NextSeq Trim         : 0 bp
Fingerprint Bins     : 100
Save Genome Index    : No
Max Resources        : 6 GB memory, 2 cpus, 12h time per job
Container            : docker - nfcore/atacseq:1.2.1
Output Dir           : ./results
Launch Dir           : /
Working Dir          : /nextflow/atacseq/rhl6d5529
Script Dir           : /.nextflow/assets/nf-core/atacseq
User                 : root
Config Profile       : test,azurebatch
Config Description   : Minimal test dataset to check pipeline function
Config Contact       : Venkat Malladi (@vsmalladi)
Config URL           : https://azure.microsoft.com/services/batch/
----------------------------------------------------
Uploading local `bin` scripts folder to az://nextflow/atacseq/rhl6d5529/tmp/66/bd55d79e42999df38ba04a81c3aa04/bin
[-        ] process > CHECK_DESIGN -

[-        ] process > CHECK_DESIGN [  0%] 0 of 1

[-        ] process > CHECK_DESIGN [  0%] 0 of 1
Error executing process > 'CHECK_DESIGN (design.csv)'

Caused by:
  Cannot find a matching VM image with publisher=microsoft-azure-batch; offer=centos-container; OS type=linux; verification type=verified




[58/55b7f7] process > CHECK_DESIGN (design.csv) [100%] 1 of 1, failed: 1
Error executing process > 'CHECK_DESIGN (design.csv)'

Caused by:
  Cannot find a matching VM image with publisher=microsoft-azure-batch; offer=centos-container; OS type=linux; verification type=verified

I tried looking into the source code of nextflow. I found the error to be in AzBatchService.groovy (line number below).

https://github.com/nextflow-io/nextflow/blob/0e593e6ab82880810d8139a4fe6e3c47ff69a531/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy#L442

I did some further digging in my Azure Batch account instance. Basically, I wanted to confirm if the list of supported images being received from the Azure Batch account has the one that is required for this pipeline. I could confirm that the server did indeed respond with the required image -

What could be the issue here? I remember running the exact same pipeline a few weeks back and it did work a few times. Am I missing something?

Answer 1

Just had another look through the Azure Cloud docs and think this might be relevant:

By default, Nextflow creates CentOS 8-based pool nodes, but this behavior can be customised in the pool configuration. Below the configurations for image reference/SKU combinations to select two popular systems.

• Ubuntu 20.04:

sku = "batch.node.ubuntu 20.04"
offer = "ubuntu-server-container"
publisher = "microsoft-azure-batch"

• CentOS 8 (default):

sku = "batch.node.centos 8"
offer = "centos-container"
publisher = "microsoft-azure-batch"

I think the issue here is a mismatched nodeAgentSkuId. Nextflow is expecting a CentOS 8 node agent SKU, but you have a CentOS 7 SKU. If it's not possible to change the nodeAgentSkuId somehow, the node agent SKU that Nextflow uses should be able to be overridden by adding this to your nextflow.config:

azure.batch.pools.<name>.sku = 'batch.node.centos 7'

Where <name> is the pool identifier:

azure.batch.pools.<name>.sku

Specify the ID of the Compute Node agent SKU which the pool identified with <name> supports (default: batch.node.centos 8, requires [email protected]).

https://www.nextflow.io/docs/edge/azure.html#advanced-settings