DeepSpeed: Efficient Training Scalability for Deep Learning
summary
DeepSpeed presents concrete systems: super offload overlaps optimizer work across GPU and Grace CPU with speculative validation, fine-tunes GPTOS 20B and Qwen 40B on one GH200 and Llama 70B on four GH200 at >50% MFU (~600 TFLOPS) and yields 4–5x speedups; Zenflow updates the top 1% of gradients on GPU (1% contributes ~90% of norm) to get ~5x end-to-end speedups without accuracy loss; Arctic long sequence training tiles and parallelizes sequences to push single‑GPU context from 32K to ~500K tokens (linear multi‑GPU scaling); compiler pass DeepCompile auto-applies ZeRO and profile‑guided prefetching for up to ~50% faster runs.
AI-written summary · about
chapters (10)
- 0:00DeepSpeed introduction and goals
- 1:26GPU memory wall and Zero optimization
- 3:09Scalability challenges in DeepSpeed
- 4:11Super Offload for superchip architectures
- 10:52ZenFlow asynchronous gradient updates
- 13:39Arctic long sequence training
- 18:17Sequence parallelism and attention support
- 19:01Deep Compile distributed optimizations
- 21:25DeepSpeed community contributions
- 22:38Q&A on PyTorch integration
related talks
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models
26mEfficiently serving LLMs at scale.
12mFlashNeuron: SSD-Enabled Large-Batch Training of Very Deep Neural Networks
17mZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
13mFrom Silicon to Scale: Training at Scale
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs