Fast Code Without Breaking Things
summary
Claimed 100% overall speedup in three months by rewriting the pipeline around Quattro (Ruby builds expression trees, Python workers run them). Identical subtrees get SHA1 ids and deduped via post-order traversal to achieve near-linear serialization. I/O drops use pushdown slicing so queries pull 200 MB instead of 40 GB. A crosstab rewrite parallelizes 100×100→100×1000 cells and cuts a 5-minute run to ~2 minutes. Hot results hit caches via serratus caching and single-worker sticky routing to maximize reuse.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 2:27Data Science Library
- 2:41Quatro
- 3:28Apology
- 4:11Data Structures
- 5:11Expressions
- 6:31Lists
- 7:36Architecture
- 11:06Subtrees
- 16:41Palm Tree
- 18:00Tree Roots
- 19:43Parallelization
- 19:58Crosstabs
- 22:39Parallelism
- 24:45Arrays caching
- 27:29Conclusion
- 27:45Matrix
- 29:05Assumptions
- 29:22Good Assumptions
- 30:21Closing
related talks
27mHigh Performance Architecture
42mSupercharge Your Performance with the Optimus Cirrus Platform
30mBlazingly Fast or Blazingly Hyped?
37mCausal's 1000x Spreadsheet; Performance beyond profiling
56mWrite Fast Code Like a Native
01h 09mThere's Treasure Everywhere
01h 50mUnlocking Modern CPU Power - Next-Gen C++ Optimization Techniques
47mHigh perf architecture at Stack Overflow