Dagster logo

Dagster

(0)
Pipeline orchestration

Dagster is an orchestration platform for developing, producing, and observing data assets.

Use it when

  • You want to create workflows as DAGs (Direct Acyclic Graphs).
  • You need flexibility (manual, scheduled, modify individual jobs according to schedule) when running DAGs.
  • You want to define inputs and output for individual jobs more explicitly to streamline data movement between tasks.
  • You want an easy way to create dynamic workflows.
  • You want to generate pipelines as Python code.
  • You want to automate ML workflows.
  • You want to separately abstract compute and storage.

Example stacks

Airflow + MLflow stack

Installation

pip install dagster dagit

Reviews