High-Performance and Scalable Middleware for HPC and Deep...
summary
MVAPICH2 and MVAPICH2-GDR integrate with Horovod to run ResNet50 on 1,536 GPUs and achieve about 0.42M images/sec (ImageNet training in ~4.5 minutes), device‑to‑device GPU latency around 760 ns and inter‑node GPU‑to‑GPU at ~2.18 µs on Summit/Lassen. A live deep introspection profiler surfaces MPI_Allreduce call counts and message‑size distributions so tuning yields ~30–90% speedups in user cases. A DPU effort offloads non‑blocking collectives to BlueField ARM cores giving up to ~22% microbenchmark and ~21% application gains. A scalable SCR-EXA checkpointing keeps checkpoint overhead flat via local+async global backups.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:45Overview
- 1:58Company Overview
- 3:38Valueadded products
- 5:43Performance
- 7:51GPU Direct RDMA
- 9:17Communication Kernel
- 16:03Deep introspection support
- 18:24Use cases
- 20:25User case
- 21:32Summary
- 22:06New Product
- 22:33Data Center Crossing Unit
- 27:05Experiments
- 27:58Benchmark
- 28:42Application
- 31:53SCR EXA
- 33:28Deep Learning Applications
- 42:59Conclusions
related talks
47mDesigning High-Performance Scalable Middleware for HPC, AI, & Data Science in Exascale Systems & ...
25mHigh-Performance MPI and Deep Learning with Introspection...
31mHigh-Performance MPI and Deep Learning on OpenPOWER Platform
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
30mDistributed Deep Neural Network Training using MPI on Python
29mDistributed Deep Learning with Horovod
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…