Scaling and Unifying SciKit Learn and Spark Pipelines using Ray
summary
A pipeline abstraction on Ray maps Scikit‑Learn and Spark pipelines to Ray tasks, actors and the distributed object store, using a list of object references as IO and novel AND/OR graph nodes to express barriers and fan‑out. A prototype runs a Kaggle preprocessing + models (random forest, gradient boost, decision tree) and reports 2x–3x speedup (up to 8x in some cross‑validation cases). Implementation supports Python and JVM interop (RayDB), immutable pipeline edits, and early‑stop/AutoML exploration.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:56Why pipelines
- 1:56Questions
- 2:43What is the focus
- 7:44Ideas
- 10:30Teaser
- 14:50Whats next
related talks
24mBuilding Complex Data Analytics Pipelines with Ray
01h 31mIntroduction to Ray for distributed and machine learning applications in Python
29mScaling ML Pipelines for Autonomous Driving with Ray
39mScaling Python for Machine Learning: Beyond Data Parallelism
03mRay: A Distributed Execution Framework for Emerging AI Applications
26mUsing Ray On Large-scale Applications at Ant Group
24mRay: A System for High-performance, Distributed ML Applications
22mRay, a Unified Distributed Framework for the Modern AI Stack