Comparing the Different Ways to Scale Python and Pandas Code
summary
Fugue provides a concrete bridge: the transform() function runs a single Python/Pandas function on Pandas Spark or Dask by swapping an engine, and a benchmark shows Spark jobs drop from 3 hours to 0.3 hours when used to prototype. Examples show when method two avoids shuffle it becomes far faster, lazy versus eager evaluation forces use of persist, and combining small Python UDFs with SQL shrinks code and often gives the best runtime.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:19Pandas Limitations
- 2:00How To Scale Out?
- 2:52Distributed Computing Frameworks
- 4:40Reducing Barrier to Entry
- 5:22Introduction to Fugue
- 7:10Fugue Transform
- 7:32Bringing it to Spark
- 9:05The DataFrame For Tests
- 9:44Pandas Assumes Data Is Physically Together
- 12:40Pandas Assumes Data Shuffle is Cheap
- 15:59Pandas Assumes Eager Evaluation
- 16:12Eager vs Lazy Evaluation
- 19:13Expectation vs Reality
- 21:21A Spark Solution Based On Traditional SOL Syntax
- 25:09Fugue SQL
- 26:47Leveraging Python
- 28:08SQL Code Size & Execution Time
related talks
38mHow to incrementally scale existing workflows on Spark, Dask or Ray?
36mScaling data workloads using the best of both worlds: pandas and Spark
01h 27mPorting Existing Python and Pandas Code to Spark, Dask, and Ray
25mScale Data Science by Pandas API on Spark
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
32mPandas May Be Slow, but Pandas Doesn't Have to Be
40mBeyond Pandas: The Great Python Dataframe Showdown
26mLarge Scale Data Validation (with Spark and Dask)