Accelerating Design Space Exploration for LLM Training Systems with Multi-experiment...
summary
Presents Multiverse, a GPU-based LLM-training simulator that runs many independent experiments in one process using SPME and data-oriented DoD modeling, with columnar storage, a pull-based (pool) synchronization and kernel-fusion to cut CPU-GPU overhead. Claims support for single-experiment runs up to 54,000 GPUs, <3.0% discrepancy versus real training, and 43.1–73.2× CPU speedup over prior simulators; ablation shows 3.2× gain from pool sync and ~16–19% reduction from kernel fusion.
AI-written summary · about
chapters (12)
- 0:00Multi-experiment simulation intro
- 2:50Parallel experiment challenges
- 4:39Four parallel strategies
- 6:07Simulator limitations
- 6:31Introducing Multiverse
- 7:58Multiverse architecture
- 9:01Modeling SPME parallelism
- 10:05GPU optimizations
- 11:11Execution correctness
- 12:39Multiverse features
- 13:00Evaluation results
- 14:33Conclusion
related talks
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
19mNo GPU Left Behind: Scaling Online LLM Training With...
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
31mThe Evolution of Multi-GPU Inference in vLLM
20mUnlocking Performance: Harnessing LLMs To Streamline GPU Kernel Development in...
01h 18mOptimizing LLM Training and Inference Performance on GPUs
46mSpeeding Up LLM Training by Over 45%
33mAccelerating the LLM Lifecycle on the Cloud