Orchest logo


Pipeline orchestration

Orchest is a data pipeline creation tool that combines Jupyter notebooks and scripts with a visual pipeline editor.

Use it when

  • You want a visual editor to create pipelines.
  • You want to run Jupyter Notebooks as pipeline steps.
  • You want to process pipeline steps in parallel for batch data.
  • You want to use Python, R, or Julia programming languages for coding.
  • You want to define input and output for individual steps explicitly.
  • You want to schedule one-off or recurring pipeline runs.
  • You want to easily spin-up services (Postgres database, Redis, TensorBoard, etc.) to run as part of your pipelines.

Watch out

  • The self-hosted version requires a Kubernetes cluster. It is possible to run Orchest locally with minikube.
  • There is no built-in support for triggering pipeline runs based on events.

Example stacks

Airflow + MLflow stack