Netflix's Ray Platform: From Deep Learning to GenAI
summary
Netflix runs daily personalized-model training on a 24‑hour cadence and scales to pools like 800‑GPU P4 clusters plus CPU pools and FSx for high throughput. A central job scheduler routes work, supports preemption/checkpointing, queue policies and dynamic resource downgrade (e.g. 800→A10), and enables HPO via an Optuna‑backed HBO manager. Ray plus VM handles multi‑GPU partitioned LLM fine‑tuning and distributed inference for multimodal pipelines that filter top 10% aesthetic images, use CLIP/Flan/LAVa/Florence, and target petabyte‑scale datasets.
AI-written summary · about
chapters (16)
- 0:00Netflix personalization
- 1:23Model production workflow
- 2:25Netflix training platform
- 4:02Heterogeneous cluster setup
- 5:36Job scheduler architecture
- 7:52Scheduler features and operations
- 10:47HPO manager
- 12:32GenAI and LLM fine-tuning
- 15:04Multimodal datasets
- 17:04Data creation requirements
- 17:37Model roles in datasets
- 20:06Practical automation examples
- 23:52Batch inference techniques
- 27:04Optimization and dataset growth
- 29:35Dataset storage and LANCE
- 30:58Questions and answers
related talks
37mAsync Inference at Netflix: Batch and Near-Real-Time Serving
22mRay, a Unified Distributed Framework for the Modern AI Stack
35mOperating a Production-Scale Physical AI Data Platform with Ray
32mRay in Production: Engineering Lessons from Spotify's Hendrix Framework
28mScaling Multimodal Data and Reinforcement Learning with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
44mPanel: ML Platform on Ray
03mRay: A Distributed Execution Framework for Emerging AI Applications