Python Doesn’t Have to Be Slow: Speeding Up a Large-Scale Optimization Algorithm
summary
A Peer‑to‑Peer portfolio allocation system uses a greedy heuristic and tight engineering to turn an intractable integer program into a fast practical pipeline: data moved out of pandas into NumPy, hot scoring kernels compiled with Numba, and suitability evaluation run embarrassingly parallel across workers. The implemented matcher handles about one million assets in ~20 seconds, moves tens of millions of pounds for thousands of investors, and maintains the platform’s affordability/return constraints.
AI-written summary · about
chapters (8)
- 0:00OPA and portfolio optimization
- 5:13Off-the-shelf solvers fail at scale
- 10:59Custom greedy algorithm
- 15:42Pandas pitfalls and NumPy speed-ups
- 17:56Parallelizing with joblib and Numba
- 20:54Numba JIT compilation
- 29:20Benchmarking and business impact
- 30:26Key takeaways and future
related talks
45mInteractive Analysis of (Large) Financial Data Sets
40mScaling Python - Bank Edition
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
14mKeynote: Scalable Python We Can Afford
32mPandas May Be Slow, but Pandas Doesn't Have to Be
23mPerformance Python: Seven Strategies for Optimizing Your Numerical Code
43mWhy do I need to know Python- I'm a pandas user
55mDerivatives Analytics with Python & Numpy