MPI Meets Python: How to Use MPI in the Python Universe
summary
Explains how to run MPI in Python using mpi4py and MVAPICH2 with concrete primitives: MPI_Send/MPI_Recv, MPI_Bcast, MPI_Reduce and MPI_Allreduce. Shows a training example splitting 100 images into five machines (10-step batches) and uses elementwise MPI_Allreduce to average gradients so training scales and ends with one synchronized model, claiming roughly a 5x speedup and noting MVAPICH2 is used on Top500 systems.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:28What is MPI
- 1:13Pointtopoint communication
- 1:58Python MPI semantics
- 3:03MPI collective communication
- 3:42MPI Broadcast
- 4:40MPI Reduce
- 5:13MPI Reducing
- 5:39MPI H2 Overview
related talks
28mWorking with multiple hands: Parallel and Distributed Programming
49mWorking with Multiple Hands
16mVisualization and Analysis of Large Scale Datasets with Python
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
16mVisualization and Analysis of Large Scale Datasets with Python
33mLet's Run Python on a Supercomputer!
26mPython + MPP Database = Large Scale AI/ML Projects in Production Faster
49mPython for High Performance and Scientific Computing