Use it when
- You want a workflow management system specializing in managing computational data analysis for fields such as Bioinformatics, Imaging, Physics, and Chemistry.
- You need a framework that supports domain-specific languages (DSL).
- You need workflows that are highly portable and are computing platform agnostic.
- You need workflows that are reproducible in different computing platforms.
- You want to support containerization.
- You want to use Kubernetes.
- You want to execute tasks in parallel efficiently.
- You want continuous checkpoints to allow workflows to resume from the last executed step.
Watch out
- Nextflow is geared towards highly specific scientific computational workflows. It can be adapted for more general ML workflows.
- Bash may complicate using Nextflow for general ML workflows.
Example stacks
Airflow + MLflow stack
Installation
curl -s https://get.nextflow.io | bash
chmod +x nextflow
sudo install nextflow /usr/local/bin/