Offloading-Efficient Sparse AI Systems
summary
Built a system called Infinity that uses request-level tracing to predict and prefetch only the experts actually used during inference, showing that four to six experts cover >90% of token traffic per layer and that under typical workloads less than 20% of parameters activate. Infinity overlaps prefetching with execution, leverages host-memory storage of experts and beats generic CUDA UVM or Brainstorm-style offloading: an A5000 (20% of H100 memory) serves large MoE layers with similar latency while using about 4x less GPU capacity.
AI-written summary · about
chapters (12)
- 0:00Efficient sparsity in AI
- 1:24Rising model sizes overhead
- 3:12Mixture of Experts explained
- 6:28Memory challenges MoE
- 8:13Efficient offloading in MoE
- 9:18Selective activation patterns
- 11:47Industry trends sustainability
- 13:14Request-level activation discovery
- 14:40System design Infinity
- 15:46Performance results efficiency
- 16:07MoE systems opportunities challenges
- 16:33Question and answer session
related talks
26mAmazingly Fast and Incredibly Scalable Inference...
15mUSHER: Holistic Interference Avoidance for Resource Optimized ML Inference
22mOptimizing Large MoE Inference on NVIDIA Blackwell: NVFP4, ADP, and DualPipe Strat...
35mUtilization Is the Key to Efficiency: What It Takes to Run...
15mECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native...
20mFireworks AI: Go from Hype to High - Value AI
20mPopFetcher: Towards Accelerated Mixture-of-Experts Training Via Popularity...
26mEfficiently serving LLMs at scale.