Use it when
- You want to serve PyTorch models and do not need a framework-agnostic serving tool.
- You want integration with popular tools such as KServe, Kubeflow, MLflow, Sagemaker, and Vertex AI.
- You want REST and gRPC support for batch inference.
- You want to version and scale your models.
- You want support for exporting metrics to Prometheus.
- You want to serve an ensemble of PyTorch models and Python functions executed as a DAG defined by a workflow.
Example stacks
Airflow + MLflow stack
Installation
git clone https://github.com/pytorch/serve.git
cd serve
python ./ts_scripts/install_dependencies.py --cuda=cu111
pip install torchserve torch-model-archiver torch-workflow-archiver