Speed up Python data processing with vectorization
summary
Vectorization means using array-backed APIs to bypass per-item Python overhead and get order-of-magnitude speedups. Using NumPy and Pandas, a Python for loop summing 30 million integers takes ~2 s while numpy.sum takes ~20 ms — about 100× faster. Store data in typed arrays, prefer bulk APIs over Python iteration or builtins like sum or map, and confirm the function iterates internally rather than calling back into Python per item.
AI-written summary · about
chapters (7)
- 0:00Python vectorization intro
- 4:40Why Python is slow
- 11:25Vectorization bypasses overhead
- 16:46Defining vectorized APIs
- 18:31Fast data processing tips
- 23:05Summary
- 23:50Q&A
related talks
27mSpeed up Python with hardware accelerators
28mWhy Python Loops Are "Slow": A Gentle Introduction to Interpretation, Bytecode, and Fast Code
29mMaking Pandas Fly
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
30m5 Steps to Speed Up Your Data-Analysis on a Single Core
45mPython for Speed, Scale, and Science
30mHighLoad Python SIMD, GPU, and Horizontal Scaling Looking into Silicon
42mIntuition vs. Reality: Surprising Truths in Python Performance