Scaling ML Pipelines for Autonomous Driving with Ray
summary
Motional reworks end-to-end ML with Ray, replacing weekly reloads with a pipeline that reduced data delivery latency from weeks to hours by streaming and Ray Data, implementing an in‑house distributed join and avoiding costly copies; an evaluation engine on Ray Data and GPUs and an in‑house metric library yields a 100X speedup for eval and moves IOU-style geometry to GPU; a dynamic ML‑based curation loop (top ~10% global selection via a holdout proxy) runs asynchronously with training on heterogeneous Ray clusters.
AI-written summary · about
chapters (6)
- 0:00Pipeline overview
- 1:00Training data preparation
- 7:38Model evaluation framework
- 13:46Dynamic data curation
- 23:36Key takeaways
- 25:15Q&A session
related talks
28mScaling Multimodal Data and Reinforcement Learning with Ray
23mHeterogeneous Pipelines as the Unlock for ML at Scale
35mOperating a Production-Scale Physical AI Data Platform with Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
44mPanel: ML Platform on Ray
26mA Growing Ecosystem of Scalable ML Libraries on Ray
17mScaling and Unifying SciKit Learn and Spark Pipelines using Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery