Scaling Python for Machine Learning: Beyond Data Parallelism
summary
Compares Spark, Dask, and Ray for scaling ML, showing that data parallelism breaks down for stateful work like model weights and reinforcement learning and that the actor model handles persistent state and message passing. Demonstrates a working Ray-based satellite backend (SpaceBeaver) using class-based Ray actors with max_restarts=-1 and shows recovery must be implemented (S3 or DB persistence). Notes Dask lazy evaluation vs Ray eager scheduling, that Dask actors have weak fault tolerance, and Spark does not expose actors.
AI-written summary · about
chapters (14)
- 0:00Intro
- 4:16Probable (relevant) biases
- 6:14Quick refresher on data parallelism
- 11:52What do (distributed) tasks look like?
- 12:49Dask distributed tasks
- 13:42Ray distributed tasks
- 14:03How are they different & same?
- 18:41Task fault tolerance
- 21:53Does Spark have tasks & actors?
- 27:39Ray Diagram
- 31:33Ray actor fault tolerance
- 33:43What's up with Ray & Netflix?
- 34:39Dask actor fault tolerance
- 37:28Outro
related talks
17mScaling and Unifying SciKit Learn and Spark Pipelines using Ray
26mRay: A System for High-performance, Distributed Python Applications
01h 30mDistributed Python with Ray-Hands on with the Ray 2.0 APIs for scaling Python Workloads
24mRay: A System for High-performance, Distributed ML Applications
01h 31mIntroduction to Ray for distributed and machine learning applications in Python
57mIntroduction to Ray for distributed and machine learning applications in Python
15mRay: A System for Distributed Applications
02h 51mDistributed Python with Ray Hands on with the Ray Core APIs