Porting Existing Python and Pandas Code to Spark, Dask, and Ray
summary
Porting Python/Pandas code uses Fugue to run the same functions on Spark, Dask, and Ray by calling a single transform interface, requiring an explicit schema to avoid expensive inference, and specifying partition by id pre-sort by date for correct time‑series diffs. The demo shows cloudpickle job serialization to submit one model per partition, recommends single‑threaded ML workers (n_jobs=1), and offers an even partitioning algorithm to avoid straggler executors.
AI-written summary · about
chapters (7)
- 0:00Welcome to Fugue
- 10:44Why distributed computing matters
- 25:48What Fugue does
- 36:59Decoupling logic and execution
- 55:12Schema best practices
- 1:08:37Mastering partitioning
- 1:26:37Recap and next steps
related talks
31mComparing the Different Ways to Scale Python and Pandas Code
39mScaling Python for Machine Learning: Beyond Data Parallelism
28mAdapting from Spark to Dask: what to expect
24mPySpark - Data processing in Python on top of Apache Spark.
01h 26mDask tutorial
29mLearning to Scale Data Science, Machine Learning, and Pandas with Ray and Modin
25mScale Data Science by Pandas API on Spark
50mBig Data Processing and Data Science Environment in Python