Scaling Interactive Pandas Workflows with Modin
summary
Modin is a drop-in middle layer for pandas that speeds up interactive workflows by changing one import import modin.pandas as pd and running on Ray with block (two‑axis) partitioning and a partition manager that reshapes partitions per operation. Benchmarks show read_csv 800MB: 29.5s → 7.6s (~4x) and near-linear scaling on many cores; the project implements supports ~71% API covering ~93% usage, defaults back to pandas for unimplemented ops, and notes indexing and some notebook demo instability as current rough edges.
AI-written summary · about
chapters (20)
- 0:00Modin introduction
- 1:45Data science tooling landscape
- 4:17Small vs large analytics
- 5:03Pandas acceleration
- 6:30Data partitioning
- 7:17Read_csv and group_by gains
- 8:48Interactive workflows
- 9:51Scaling from laptops to large servers
- 11:16Pandas API support
- 13:52Fallback to pandas
- 15:39Modin architecture
- 18:08Ray’s role in Modin
- 20:02Live demo
- 23:44Modin benefits
- 26:19Q&A interactivity
- 28:09Q&A scaling
- 30:10Q&A memory
- 31:19Modin vs Dask
- 33:52Handling large data
- 35:35Q&A on where Modin breaks most often
related talks
23mSupercharging your pandas workflows with Modin
29mLearning to Scale Data Science, Machine Learning, and Pandas with Ray and Modin
24mHow to maximally parallelize the entire pandas API
01h 17mPandas from the Inside / "Big Pandas"
44mHigh Perfomance Pandas
32mPandas May Be Slow, but Pandas Doesn't Have to Be
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
35mFast Pandas Performance Tricks You Wish You Knew Earlier