支持平台
cromwell安装后可以直接进行单机,单节点的任务投递。1
java -jar cromwell.jar run pipeline.wdl -i pipeline.json
但是 cromwell 不仅支持本地计算机任务调度,同时支持集群/云计算作业管理系统。cromwell针对包含 SGE、AWS、Docker 在内的等多种平台和容器均提供了配置文件模板,可以通过简单的配置实现基于各平台的调度。
Cloud Providers
- AWS: Amazon Web Services (documentation)
- TES: is a backend that submits jobs to a server with protocol defined by GA4GH (documentation)
- PAPIv2: Google Pipelines API backend (version 2!) (documentation)
Containers
- Docker: an example backend that only runs workflows with docker in every command
- Singularity: run Singularity containers locally (documentation)
- Singularity+Slurm: An example using Singularity with SLURM (documentation)
- TESK is the same, but intended for Kubernetes. See the TES docs at the bottom.
- udocker: to interact with udocker locally documentation
- udocker+Slurm: to interact with udocker on SLURM (documentation)
Workflow Managers
- HtCondor: a workload manager at UW-Madison (documentation)
- LSF: the Platform Load Sharing Facility backend (documentation)
- SGE: a backend for Sungrid Engine (documentation)
- slurm: SLURM workload manager (documentation)
集群配置
SGE
官方针对不同的集群/云作业管理系统提供了相关的配置文件(https://github.com/broadinstitute/cromwell/tree/develop/cromwell.example.backends),但是本质都是讲调度命令嵌入其中。**作业调度系统的配置文件并非完整的配置文件,必须添加到 [cromwell.examples.conf](https://github.com/broadinstitute/cromwell/blob/develop/cromwell.example.backends/cromwell.examples.confhttps://github.com/broadinstitute/cromwell/blob/develop/cromwell.example.backends/cromwell.examples.conf) 的 backend 部分才可以正常工作**
1 | # cromwell.sge.config |
使用sge进行任务投递1
java -Dconfig.file=cromwell.examples.conf -jar /home/liubo4/software/cromwell-84.jar run helloword.wdl -i helloword.wdl.input.json
SGE&docker
1 | # cromwell.sge.docker.config |
reference
Cromwell ReadtheDoc SGE
Cromwell Example Backends
[https://blog.csdn.net/tanzuozhev/article/details/120629170]