From Benchmark #1 to Production
summary
Nebius claims production-ready inference wins: a Kernel Design Agent that auto-generates and verifies kernels yields an ~8x speedup on a deepsea sparse-attention kernel; a speculative-decoding tweak called SlimSpec uses a low-rank LM-head bottleneck (1/8 rank) to get ≈5x LM-head speed and ~10% end-to-end gain while keeping 99%+ token acceptance; disaggregated prefill/decode quantization uses high-precision prefill and low-precision decode (NVFP4 on Blackwell cited) to reach ~2–2.5x decode speed and ~2.3x end-to-end; all deployed on Nebius Token Factory.
AI-written summary · about
chapters (9)
- 0:00Introduction and background
- 1:31Benchmark vs production challenges
- 3:19Kernel design agent explained
- 5:27Spec decoding optimizations
- 9:02Disaggregated quantization
- 12:32Hardware co-design approach
- 13:55Lessons from production deployment
- 14:38Recap of techniques
- 15:18Closing and Q&A
related talks
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
38mOptimizing LLM Inference for the Rest of Us
36mPanel Discussion
26mEfficiently serving LLMs at scale.
30mAccelerated LLM Inference with Anyscale
24mLLM Inference Optimization From Simple To Sophisticated
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
34mOptimizing AI Inference with ML Compilers & Hardware