WaferLLM: Large Language Model Inference at Wafer Scale
summary
WaferLLM adapts LLM inference to wafer-scale chips using a PLMR model that guides placement and communication, implements MeshGEMM and MeshGEMV operators, and uses a tree-based reduce plus a shift-based KV cache to avoid hotspotting. Deployed on a Cerebras WSE2, it reaches million-scale parallelism, reports up to 200× higher accelerator utilization, GEMV 606× faster and 16× more energy-efficient than an A100, and 10–20× full-inference speedups versus A100 clusters.
AI-written summary · about
chapters (13)
- 0:00Introduction and motivation
- 0:45Limits of off-chip scaling
- 2:27Wafer-scale integration
- 3:30Evolution of wafer-scale chips
- 5:36Wafer-scale performance gains
- 6:54Software challenges for hardware
- 7:53The PLMR model
- 10:30WaferLLM system architecture
- 11:52Optimizing decode parallelism
- 12:49Efficient communication methods
- 13:43KV-cache management strategies
- 14:43Performance evaluation
- 16:03Future outlook and conclusion
related talks
11mKeynote: Building a Large Model Inference Platform for Heterogeneous Chinese Chips Based on VLLM
21mScaling LLM Inference with vLLM
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
40mOpen Source LLMs in the Cloud: Scalable Solutions
31mThe Evolution of Multi-GPU Inference in vLLM
33mllm-d: Kubernetes Native Distributed Inferencing
29mThe Evolution of LLMs: From Imitation to Reasoning
23mNew Pattern for Sailing Multi-host LLM Inference