Optimizing AI Inference with ML Compilers & Hardware
summary
Argument: ML compilers (TVM, MLIR, Triton) and hardware decisions determine whether LLMs are practical or too expensive — estimates show up to 90% of total cost can be spent on inference. Techniques demonstrated include quantization and pruning to shrink models, KV cache and speculative decoding to cut decode latency, and custom in‑memory accelerators that move LLaMA‑270B from ~240 GB to usable rates; examples show a 37,000× kernel speedup claim and hardware benchmarks like 136 vs 2,000 tokens/sec.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:11ML compilers for AI
- 5:14AI hardware platforms
- 7:01ML compiler complexity
- 10:33ML compiler structure
- 13:02ML compilers TVM Triton
- 16:14AI inference optimizations
- 19:05LLM size challenges
- 20:33Transformer metrics
- 22:38LLM prefill decode
- 24:24KV cache speculative
- 25:49Hardware-software co-design
- 26:53AI accelerators evolution
- 28:20LLM memory bottlenecks
- 29:03In-memory AI compute
- 30:51AI hardware breakthroughs
- 31:55Agentic AI scaling
- 32:58Heterogeneous computing NPUs
- 33:41AI models hardware future
related talks
24mCost and Performance Optimization of LLM Inferencing.
24mLLM Inference Optimization From Simple To Sophisticated
30mLLM Inference Arithmetics: the Theory behind Model Serving
28mThe AI Tipping Point
38mOptimizing LLM Inference for the Rest of Us
54mProgramming Language Design and Implementation for the Machine Learning Era: A Personal(…)
41mSub-Linear Algorithms meets Large Language Models
30mAccelerated LLM Inference with Anyscale