Use it when
- You want to create workflows as DAGs (Direct Acyclic Graphs).
- You need flexibility (manual, scheduled, modify individual jobs according to schedule) when running DAGs.
- You want to define inputs and output for individual jobs more explicitly to streamline data movement between tasks.
- You want an easy way to create dynamic workflows.
- You want to generate pipelines as Python code.
- You want to automate ML workflows.
- You want to separately abstract compute and storage.
Airflow + MLflow stack
pip install dagster dagit