How to Tune your Python Analysis Pipeline: A Profiler Guide
summary
A concrete data‑analysis pipeline shows profiling moving from guessing to fixes: initial run takes ~20s, switching JSON text reads to binary/HDF5 and using NumPy cuts wall time to about 3s and lowers peak memory from ~40 to ~23. Timing and memory bottlenecks come from data loading and list/dict-heavy reductions. Workflow uses cProfile, visualizes with SnakeViz, samples with PySpy and measures CPU+memory with Scalene, demonstrating practical, reproducible gains.
AI-written summary · about
chapters (16)
- 0:00Introduction to Python profilers
- 1:52Profiling vs benchmarking
- 3:56Instrumenting vs sampling profilers
- 7:27Example pipeline setup
- 9:14Benchmarking with time command
- 9:56Using cProfile for timing
- 11:44Visualizing with SnakeViz
- 13:30Optimizing data loading
- 14:13Sampling with PySpy
- 15:59Analyzing combined stats
- 16:20Introducing Scalene profiler
- 17:47Memory profiling basics
- 18:54Optimizing memory usage
- 20:44Using memory profiler
- 21:47Summary of Python profilers
- 24:55Q&A session
related talks
30m5 Steps to Speed Up Your Data-Analysis on a Single Core
01h 31mDebugging and profiling techniques
37mTuning Python applications can dramatically increase performance
22mExperiences making CPU-bound tasks run much faster
28m"Performant Python"
37mMake it Work, Make it Right, Make it Fast Debugging and Profiling in Dask
02h 33mMeasure, don't Guess - How to find out if and where to optimize
30mSpeed Up Your Data Processing