Scheduler and Sharding Considerations for Network Efficiency
summary
Describes how MAST and topology-aware placement map Llama3 parallelisms onto a 24,000 GPU network to cut cross‑hop traffic, using rank assignment and topological allocation constraints so tensor parallel groups stay within racks and pipeline groups within AI zones. Claims concrete gains: Llama3 405B needs ~700 TB and 16K GPUs and tuning yields 2.5x GPU utilization. Explains practical costs: buffer sprawl for spare hosts (example: 50 racks stranded if one host per rack fails) and scheduler-level tradeoffs.
AI-written summary · about
chapters (5)
- 0:00Network efficiency challenges
- 2:31Parallelism and model network co-design
- 9:56Scheduling at scale for ML training
- 17:00Infra and model co-design
- 19:52Key takeaways
related talks
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
31mThe Evolution of Multi-GPU Inference in vLLM