Optimize Your AI Cloud Infrastructure: A Hardware Perspective
summary
Benchmarks run MLPerf CNN/RNN/Transformer models on an H100 cluster using Megatron/DeepSpeed and measure the impact of POD scheduler, Device Plugin, GPU/NUMA topology and the ROCE/NCCL stack. Experiments show PCIe/ACS/ATS misconfiguration can drop effective link rate to ~85 Gbps instead of expected 300–500 Gb/s, creating multi‑node bottlenecks. Tuning network offload, NVLink layout and placement raises MFU (model FLOPS utilization) and overall training throughput across nodes.
AI-written summary · about
chapters (13)
- 0:00AI cloud infrastructure intro
- 1:08AI models and infrastructure growth
- 3:25Single-GPU to multi-GPU HPC
- 6:36Why AI cloud matters
- 11:38Hardware bottlenecks in AI
- 16:20GPU memory and interconnect speeds
- 21:37Key components to optimize
- 26:35Multi-node training splits
- 30:10MLPerf benchmarks for training
- 32:25Hardware optimization principles
- 34:12Real-world GPU lessons
- 36:16Emerging AI hardware trends
- 37:24Q&A on comms costs
related talks
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
34mNetworking Optimizations for Multi-Node Deep Learning on Kubernetes
19mAI/ML Networking Challenges: The Fast and the Finicky!
41mBeyond Uptime: Revolutionizing Fintech Reliability
34mNetworking Optimizations for Multi-Node Deep Learning on Kubernetes
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
46mWhen AI Meets High Performance Computing