Let's Run Python on a Supercomputer!
summary
Shows how plain Python scales on an HPC stack: JIT with Numba yields 30x with Numba on a single core, using multiprocessing across an 8-core node gives 6x on 8 cores, and naive MPI across 100 ranks only gives 1.5x with 100 MPI workers because of serial overheads. Offloading to a GPU via Numba.cuda adds about 4x on top of the CPU run. Uses Slurm for scheduling, Singularity or Conda for deps, and cloud clusters/spot instances for cheap, on-demand nodes.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 5:57Supercomputer programming
- 15:44GPUs
- 23:44Small jobs
- 24:59GPU implementation
- 25:56Cloud performance
- 27:24How much time do you need
- 28:34Questions
- 30:18Optimization
related talks
37mAccelerating Scientific Computing in Python with Numba: A Real-World Example
22mSupercharge Scientific Computing in Python with Numba
30mHighLoad Python SIMD, GPU, and Horizontal Scaling Looking into Silicon
45mPython for Speed, Scale, and Science
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
43mKEYNOTE: Scaling Out PyData
02h 28mGPU Accelerated Python