azure - Nextflow 与 Azure Batch - 找不到匹配的 VM 镜像

Question

尝试使用 Azure Batch (NF-Core) 设置 Nextflow 时，出现以下错误。我在多个工作流程(sarek、ataseq 等)上尝试过此操作，但遇到了相同的错误 -

N E X T F L O W  ~  version 22.04.0
Pulling nf-core/atacseq ...
 downloaded from https://github.com/nf-core/atacseq.git
Launching `https://github.com/nf-core/atacseq` [rhl6d5529] DSL1 - revision: 1b3a832db5 [1.2.1]
Downloading plugin <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="1a747c377b606f687f5a2a342b29342b" rel="noreferrer noopener nofollow">[email protected]</a>
----------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/atacseq v1.2.1
----------------------------------------------------

Run Name             : rhl6d5529
Data Type            : Paired-End
Design File          : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/design.csv
Genome               : Not supplied
Fasta File           : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/reference/genome.fa
GTF File             : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/reference/genes.gtf
Mitochondrial Contig : MT
MACS2 Genome Size    : 1.2E+7
Min Consensus Reps   : 1
MACS2 Narrow Peaks   : No
MACS2 Broad Cutoff   : 0.1
Trim R1              : 0 bp
Trim R2              : 0 bp
Trim 3' R1           : 0 bp
Trim 3' R2           : 0 bp
NextSeq Trim         : 0 bp
Fingerprint Bins     : 100
Save Genome Index    : No
Max Resources        : 6 GB memory, 2 cpus, 12h time per job
Container            : docker - nfcore/atacseq:1.2.1
Output Dir           : ./results
Launch Dir           : /
Working Dir          : /nextflow/atacseq/rhl6d5529
Script Dir           : /.nextflow/assets/nf-core/atacseq
User                 : root
Config Profile       : test,azurebatch
Config Description   : Minimal test dataset to check pipeline function
Config Contact       : Venkat Malladi (@vsmalladi)
Config URL           : https://azure.microsoft.com/services/batch/
----------------------------------------------------
Uploading local `bin` scripts folder to az://nextflow/atacseq/rhl6d5529/tmp/66/bd55d79e42999df38ba04a81c3aa04/bin
[-        ] process > CHECK_DESIGN -

[-        ] process > CHECK_DESIGN [  0%] 0 of 1

[-        ] process > CHECK_DESIGN [  0%] 0 of 1
Error executing process > 'CHECK_DESIGN (design.csv)'

Caused by:
  Cannot find a matching VM image with publisher=microsoft-azure-batch; offer=centos-container; OS type=linux; verification type=verified




[58/55b7f7] process > CHECK_DESIGN (design.csv) [100%] 1 of 1, failed: 1
Error executing process > 'CHECK_DESIGN (design.csv)'

Caused by:
  Cannot find a matching VM image with publisher=microsoft-azure-batch; offer=centos-container; OS type=linux; verification type=verified

我尝试查看nextflow的源代码。我发现错误位于 AzBatchService.groovy(下面的行号)中。

https://github.com/nextflow-io/nextflow/blob/0e593e6ab82880810d8139a4fe6e3c47ff69a531/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy#L442

我在我的 Azure Batch 帐户实例中做了一些进一步的挖掘。基本上，我想确认从 Azure Batch 帐户收到的受支持图像列表是否包含该管道所需的图像。我可以确认服务器确实响应了所需的图像 -

这里可能出现什么问题？我记得几周前运行了完全相同的管道，并且确实工作了几次。我错过了什么吗？

Answer 1

刚刚又看了一遍Azure Cloud文档并认为这可能相关:

By default, Nextflow creates CentOS 8-based pool nodes, but thisbehavior can be customised in the pool configuration. Below theconfigurations for image reference/SKU combinations to select twopopular systems.

• Ubuntu 20.04:

sku = "batch.node.ubuntu 20.04"
offer = "ubuntu-server-container"
publisher = "microsoft-azure-batch"

• CentOS 8 (default):

sku = "batch.node.centos 8"
offer = "centos-container"
publisher = "microsoft-azure-batch"

我认为这里的问题是不匹配的 nodeAgentSkuId 。 Nextflow 需要 CentOS 8 节点代理 SKU，但您有 CentOS 7 SKU。如果无法更改 nodeAgentSkuId不知何故，Nextflow 使用的节点代理 SKU 应该能够通过将其添加到您的 nextflow.config 来覆盖。 :

azure.batch.pools.<name>.sku = 'batch.node.centos 7'

哪里<name>是池标识符:

azure.batch.pools.<name>.sku

Specify the ID of the Compute Node agent SKU which the pool identified with <name> supports (default: batch.node.centos 8, requires <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="026c642f637877706742322c33332c32" rel="noreferrer noopener nofollow">[email protected]</a>).

https://www.nextflow.io/docs/edge/azure.html#advanced-settings