Experiences making CPU-bound tasks run much faster
summary
Profiling with cProfile and line_profiler finds hot loops, then applied PyPy, NumPy vectorization, numexpr, C extensions and GPU kernels to the hot spots. One workflow shrinks a 234 s job to ~90 s then ~81 s and finally ~12 s after adding C code; another case goes 35 s (pure Python) → 10 s (PyPy) → 36 s (NumPy) → 19 s (numexpr) → 0.6 s (C) → 0.47 s (Shed Skin) → 0.07 s on a GTX-class CUDA GPU.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:21Goals
- 0:44Resources
- 1:40About me
- 2:43What we are using
- 7:34Benchmarks
- 9:45Physics
- 13:35Slow implementation
- 20:15Line profiling
related talks
57mPython and PyPy performance (not) for dummies
37mTuning Python applications can dramatically increase performance
01h 21mAccelerating Python With C++
32mProfiling at the speed of light
30mHow to Tune your Python Analysis Pipeline: A Profiler Guide
01h 33mPower up your work with compiling and profiling
33mEasy wins with Cython: fast and multi-core
02h 28mGPU Accelerated Python