Tailor-Made Inference at Scale
summary
A configurable inference engine breaks inference into swappable atomic components and an agent-driven search composes stacks from vLLM and TensorRT-LLM backends, custom kernels and a YAML config schema; an experimentation ledger (EBS/ABS) records benchmarks so the agent runs hypotheses, finds the best config, and predicts cost vs performance tradeoffs including autoscaling per workload and prompt-level tuning for data privacy. Real production surprises get caught by shadow load testing, with the same model measured from 180–780 tokens/sec depending on configuration and kernel choice.
AI-written summary · about
chapters (7)
- 0:00Introduction and setup
- 1:08Open source models in production
- 3:17Customizing inference stacks
- 5:22Benchmarking configurations
- 8:36Shadow load testing
- 10:21Compute strategy and scaling
- 12:28Q&A and wrap-up
related talks
19mServerless inferencing: an infrastructure point of view
30mLLM Inference Arithmetics: the Theory behind Model Serving
39mAccelerating Open-Source RL and Agentic Inference with vLLM
28mNavigating LLM deployment: Tips, tricks, and techniques
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
24mLLM Inference Optimization From Simple To Sophisticated
38mOptimizing LLM Inference for the Rest of Us