How Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
summary
ByteDance builds a Ray-based pipeline (Ray Core, Ray Data, Ray Serve) to process petabyte-scale audio/video for generation models, using FFMPEG for encode/decode and packing clips into pack files for training to avoid millions of small object reads. Early design serialized huge binaries to the object store; the fix is to fuse operators into single actor to stream work and hit high throughput, and to extend Ray Data with task reassignment with linkage table (max_restarts=0) to recompute lost outputs on preemptible Kubernetes nodes. Profiling with the Ray dashboard validates scaling to thousands of workers.
AI-written summary · about
chapters (10)
- 0:00Introduction and background
- 2:22Audio processing architecture
- 3:33DAG design for data flows
- 3:55Challenges in data processing
- 5:16Transitioning to Ray
- 7:11Video data processing
- 15:56Lessons and Ray advantages
- 18:00Infrastructure overview
- 18:51Running on volatile pods
- 20:59Enhancements and conclusions
related talks
26mData Processing on Ray
24mBuilding Complex Data Analytics Pipelines with Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
22mLarge Scale Data Loading and Data Preprocessing with Ray
28mScaling Multimodal Data and Reinforcement Learning with Ray
31mFrom OOMs to 2x Throughput: Scaling Multimodal Data Curation on Ray
35mOperating a Production-Scale Physical AI Data Platform with Ray
32mRay in Production: Engineering Lessons from Spotify's Hendrix Framework