Chasing Pandas
summary
Company builds a translation layer called Quattro that serializes Ruby expression trees, sends them via Redis to Python workers running pandas, and returns results with only microseconds of overhead per node; benchmarks show Daru about two orders of magnitude slower while Quattro matches pandas for typical survey workloads (hundreds of thousands of rows, 50–100 columns). Features include single-worker transactions, subexpression caching, tree rewrites and index partitioning; RAM needs (5–10x dataset) remain the main limiter, Arrow and Rubik noted as next steps.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:30Why Ruby for data
- 5:23What is pandas
- 7:11Quattro solution
- 10:52Diary gem overview
- 13:44Demo setup
- 15:58Diary demo
- 17:28Quattro demo
- 19:21Performance comparison
- 21:55Performance bottlenecks
- 22:58Future improvements
- 24:45Closing thoughts
- 27:01Q&A session
related talks
31mBeyond Pandas: lightning fast in-memory dataframes with Polars
32mPandas, Polars and the DataFrame Consortium.
28mData Science in Ruby? Is it possible? Is it Fast? Should we use it?
42mDevelopment of Data Science Ecosystem for Ruby
24mPandas on Steroids
28mIdiomatic Pandas
31mBringing the Jewels of the Python World to Scala with Spark
42mii hidden gems