Nvidia Triton Inference Server: building production ML without developers
summary
A practical case study of building a production ML inference platform using NVIDIA Triton Inference Server and Kubernetes. Covers problems encountered with Seldon (and reasons for abandoning it), and how canary deployments of models were implemented using Istio.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 6:48Choosing Triton over Seldon
- 11:22Five platform elements
- 14:35Auto-scaling resources
- 23:32Inference graphs and Ray
- 27:06Triton optimization
- 30:36Building UI without developers
- 36:40Summary and future plans
- 50:14Q&A
related talks
25mMoving Inference to Triton Servers
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve
22mDeploying computer vision models at scale devops-free
26mBuilding a Cloud Native Edge Platform for Accelerating AI Inference
10mBuilding Machine Learning Inference Through Knative Serverless...
29mA Scalable Platform for Training and Inference Using Kubeflow
18mBeyond Inference: Bringing ML into Production
17mDevOps for Private AI: A Scalable Platform Using Kubeflow