Inference Deployments and Comms Implication
summary
Introduces DDA flat and DDA tree all‑reduce variants that trade extra data movement for fewer steps and report up to 50% speedup on small messages (<5 MB), translating to ~4% end‑to‑end prefill gain and ~10% decoding gain; shows contest parallelism that claims sub‑minute inference for 1M and 10M token contexts on H100 setups and near‑linear scaling; deploys persistent alltoall via a custom transport (cand) cutting alltoall latency ~22–50%; demonstrates 10–30% microbenchmark gains from compute+communication fusion and advocates device‑initiated fused kernels.
AI-written summary · about
chapters (7)
- 0:00Inference deployments overview
- 3:18Tensor parallelism
- 8:52Optimizing all-reduce with DDA
- 9:14Context parallelism
- 12:48Expert parallelism
- 18:44System improvements
- 21:13Conclusion & Q&A
related talks
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
30mAccelerated LLM Inference with Anyscale
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
16mFrom Benchmark #1 to Production
34mOptimizing AI Inference with ML Compilers & Hardware
57mPervasive and Sustainable AI with Adaptive Computing
20mHacking the Inference Pareto Frontier
26mEfficiently serving LLMs at scale.