Optimize first, parallelize second: a better path to faster data processing
summary
If code is slow, the decisive details are what actually changes performance: the measured speedup numbers, the algorithmic complexity fixed, the architecture or library that removes wasted work, and whether a tuned implementation runs reliably faster. Prioritize optimize first, fix hot spots with profiling and better algorithms or a different library (pandas vs Polars, NumPy/SciPy, scikit-image), then consider parallelize second only if single-core limits remain.
AI-written summary · about
chapters (17)
- 0:00Software performance priorities
- 1:12Data processing performance focus
- 3:01Slow software problems
- 5:11Faster CPUs won't fix issues
- 6:35Parallelism as solution
- 8:41Optimization overlooked
- 10:07Parallelism limits
- 11:31Parallelism costs unchanged
- 14:20Optimization vs parallelism costs
- 16:06Performance optimization levels
- 17:10Pandas vs Polars differences
- 20:18Avoid quadratic algorithms
- 22:03Image processing with LUTs
- 24:07Tuning for CPU features
- 25:30Optimize before parallelize
- 27:36Closing remarks
- 27:58Q&A on Python optimization
related talks
30mSpeed Up Your Data Processing
19mPython - best practices
01h 17mPandas from the Inside / "Big Pandas"
31mPolars Plugins: how you (yes, you!) can extend Polars Dataframes
01h 02mPandas from the inside
19mProfiling Your Code: 5 Tips to Significantly Boost Performance
16mImprove application and code quality without overengineering
57mParallelize your code, one step at a time