Building Complex Data Analytics Pipelines with Ray
summary
Describes using Ray to convert Python preprocessing, document-level entity extraction, corpus-level clustering and entity linking into remote tasks so data scientists run end-to-end experiments themselves. Claims a small team of 2.5 data scientists and engineers rewrites little code and achieves reduced pipeline runtime by over 50% after a few days of work. Relies on Ray’s shared-memory object store and targets on‑prem deployments across sites with highly variable hardware. Demonstrates faster large‑scale testing and portable pipelines that actually run.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:36Ideal setup
- 4:38Problems
- 9:16Data Scientists
- 11:22Pipeline Parallelization
- 13:36Faster iteration
- 14:23Healthcare ML pipeline
- 19:33Benefits of largescale testing
- 21:13Challenges
related talks
26mData Processing on Ray
31mNext-Generation Data Science Workflows Using Ray
26mRayDP: Build Large-scale End-to-end Data Analytics and AI Pipelines Using Spark and Ray
17mScaling and Unifying SciKit Learn and Spark Pipelines using Ray
01h 31mIntroduction to Ray for distributed and machine learning applications in Python
22mRay, a Unified Distributed Framework for the Modern AI Stack
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
15mRay: A System for Distributed Applications