Accelerating Machine Learning at Scale with
summary
Takes a GPT-2 model from HuggingFace, converts and quantizes it with Optimum into ONNX conversion artifacts, runs inference locally on MLServer and deploys to Kubernetes with Seldon, and benchmarks with k6 to compare CPU and GPU. Demonstrates adaptive batching and micro-batching, shows 10x-100x speedup on GPU under batched loads, wires CI/CD via Flux for GitOps, and validates that the optimized artifact and deployment actually work at scale.
AI-written summary · about
chapters (15)
- 0:00Start
- 5:25Code Examples & Links
- 5:49Fetching the model
- 6:18Inference with Model Directly
- 7:21The Internals of "generate(...)"
- 8:32Optimizing with Optimum & ONNX
- 10:36Challenges to Scaling
- 13:09Productionising with Seldon
- 13:25Define GPT2 Model Pipeline
- 14:58Run with MLServer locally
- 17:39Seldon MLServer DEMO
- 18:26Deploy with Seldon Core
- 18:59Send inference requests
- 28:19Production Observability
- 29:01Load Testing & Monitoring
related talks
36mAccelerating High-Performance Machine Learning at Scale i...
31mOvercoming Challenges in Serving Large Language Model
42mDeploying ML at Scale with Kubernetes and TFX
34mServing ML Models at Scale with Seldon and Kubeflow
37mFrom Hours To Milliseconds: Scaling AI Inference 10x With Serverless on Kubernetes
40mScalable machine-learned model serving
28mProductionizing Machine Learning Models at Scale with Kubernetes
37mEnd-2-End Machine Learning on GCP