Data Processing on Ray
summary
Ray adds concrete systems work: locality-aware scheduling and admission control so tasks aren’t placed where inputs won’t fit; a decentralized memory manager that enforces hard object-store limits; and object spilling to disk/S3 with prioritized spill/restore to enable out‑of‑core shuffles. Integrations (Modin, Dask on Ray, Spark on Ray, Ray Dataset, Ray Tune) run on that stack, and benchmarks report running very large shuffles (reportedly ~900‑terabyte class) with improved progress and fewer pointless spill/restore cycles.
AI-written summary · about
chapters (7)
- 0:00Introduction to Ray
- 1:04Challenges in data pipelines
- 4:26Ray as a unified platform
- 10:20Requirements for data backend
- 13:29Distributed shuffle improvements
- 17:25Object spilling techniques
- 20:43Memory management and conclusion
related talks
29mDask-on-Ray: Using Dask and Ray to Analyze Petabytes of Remote Sensing Data
29mLearning to Scale Data Science, Machine Learning, and Pandas with Ray and Modin
57mIntroduction to Ray for distributed and machine learning applications in Python
01h 31mIntroduction to Ray for distributed and machine learning applications in Python
26mRay: A System for High-performance, Distributed Python Applications
39mScaling Python for Machine Learning: Beyond Data Parallelism
01h 30mDistributed Python with Ray-Hands on with the Ray 2.0 APIs for scaling Python Workloads
31mScaling Python to thousands of nodes with Ray