Accelerating High-Performance Machine Learning at Scale i...
summary
Deploys Distil GPT-2 from the Hugging Face hub, converts it to a quantized ONNX artifact with Optimum, and serves it locally with MLServer before scheduling identical runtimes on Kubernetes via Seldon Core (routing with Istio/Ambassador). A k6 load test shows GPU batching cuts inference from about 2 seconds on CPU to ~200 ms on GPU (≈10×). Repo-based delivery uses Flux/GitOps for reproducible rollouts, and multi-model serving plus adaptive batching are used to share GPU capacity.
AI-written summary · about
chapters (17)
- 0:00Start
- 3:40Code Examples & Links
- 4:07Fetching the model
- 4:49Inference with Model Directly
- 5:40The Internals of "generate(...)"
- 6:06Optimizing Model with ONNX
- 7:09Challenges to Scaling
- 10:09Run with MLServer locally
- 13:26Deploy with Seldon Core
- 13:59Send inference requests
- 15:08Continuous Delivery via GitOps
- 15:56Programmatic MLOps via Gitops
- 17:40Production Observability
- 18:44Load Testing & Monitoring
- 21:38Seldon Core on AKS DEMO
- 23:05Anatomy of Prod MLOps
- 26:29Further Resources
related talks
36mAccelerating Machine Learning at Scale with
26mEfficiently serving LLMs at scale.
34mServing ML Models at Scale with Seldon and Kubeflow
31mOvercoming Challenges in Serving Large Language Model
28mTraining Large Language Models on Kubernetes
34mDemocratizing Machine Learning on Kubernetes [I]
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
21mOptimizing Model Serving on Kubernetes With Model Streaming