Use it when
- You want a visual editor to create pipelines.
- You want to run Jupyter Notebooks as pipeline steps.
- You want to process pipeline steps in parallel for batch data.
- You want to use Python, R, or Julia programming languages for coding.
- You want to define input and output for individual steps explicitly.
- You want to schedule one-off or recurring pipeline runs.
- You want to easily spin-up services (Postgres database, Redis, TensorBoard, etc.) to run as part of your pipelines.
- The self-hosted version requires a Kubernetes cluster. It is possible to run Orchest locally with minikube.
- There is no built-in support for triggering pipeline runs based on events.
Airflow + MLflow stack