Use it when
- You want to visualize pipelines running in production.
- You want to generate pipelines as YAML code.
- You want to deploy machine learning models.
- You want to adopt containers and Kubernetes for developing and deploying distributed applications.
- You want to create workflows as DAGs (Direct Acyclic Graphs).
- You want to run each task on a separate Kubernetes pod.
- You want a workflow orchestrator that natively adds services like, secrets, roles, and volumes to Kubernetes.
- You want to configure infrastructure as YAML code.
- You want resilience to container crashes.
- You want support for both time and event-based workflow execution.
- You want autoscaling features.
- You want a workflow orchestrator that easily installs to existing Kubernetes clusters.
- Maintaining the YAML files for large projects is difficult.
- Requires expertise in Kubernetes to work safely in production workloads.
- The maintenance of an enterprise setup is complex.
Airflow + MLflow stack
kubectl create ns argo kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/master/manifests/quick-start-postgres.yaml