Accelerating Distributed MoE Training and Inference
summary
Lina is a systems layer for distributed MoE that opportunistically prioritizes expensive all-to-all token exchanges over concurrent allreduce by splitting communications into microops and tensor partitioning, enabling preemption and pipelining with expert computation; it also profiles skewed expert popularity during training to run a two-phase scheduling at inference that packs unpopular experts and allocates GPU resources to hot experts. Evaluated on an A100 testbed, Lina reports up to 1.73x training step speedup and an average 95th‑percentile 1.63x reduction in inference tail latency.
AI-written summary · about
chapters (12)
- 0:00Introduction to Lina
- 0:36Background on MoE models
- 1:23MoE in transformer models
- 3:02Distributed MoE challenges
- 4:25Analyzing communication
- 5:56Training vs. inference issues
- 7:24Resource bottlenecks
- 9:20Lina training optimization
- 11:21Lina inference optimization
- 13:20Two-phase scheduling design
- 15:59Evaluation and results
- 18:55Summary and conclusion
related talks
27mStageML: Partial Evaluation for Multi-Tenant MoE-LoRA Inference
18mSparse Checkpointing for Fast and Reliable MoE Training
46mSpeeding Up LLM Training by Over 45%
01h 18mOptimizing LLM Training and Inference Performance on GPUs
45mSymbiotic Relations between Decoupled Training, Optimization, and Federated Learning
19mML training acceleration with heterogeneous resources in Bytedance
26mMemory-Efficient LLM Inference on Edge Devices With NNTrainer
15mUSHER: Holistic Interference Avoidance for Resource Optimized ML Inference