Use it when
- You want a platform flexible workflow orchestrator deployed either on-prem, Kubernetes cluster, or as a Docker container.
- You want to define pipelines declaratively as YAML code.
- You want a data orchestrator that supports ETL and ELT pipelines.
- You want support for parallel and branched tasks in a pipeline.
- You want event-based pipeline scheduling triggered by API, a timed schedule, detection, and events.
- You want to track pipeline metrics.
- You want to use Terraform to manage cloud resources.
- You want a developer-friendly UI.
Watch out
- Production workflows require a Kubernetes cluster.
Example stacks
Airflow + MLflow stack