Introduction to Distributed ML Workloads with Ray on Kubernetes
summary
Introduces Ray core concepts (actors, tasks) and shows running Ray on Kubernetes using the KubeRay operator, translating Ray jobs into Kubernetes CRDs and pods; demonstrates scaling workers from 1 to 3 via kubectl edit, submitting jobs with "ray job submit" to localhost:8265, and viewing the dashboard on port 8265. Explains GPU scheduling, use of spot VMs for cheap autoscaling, and a working example that serves a fine-tuned Llama 2 with Gradio (Ray Serve defaults to TGI inference and integrates with LangChain).
AI-written summary · about
chapters (8)
- 0:00Introduction to Ray
- 2:21Ray Core and Libraries
- 5:27Ray on Kubernetes
- 12:51Live Demo: Scaling Ray
- 15:46Monitoring with Dashboard
- 17:19Advanced Ray Features
- 19:49LLM Serving Example
- 23:13Q&A and Conclusion
related talks
39mSailing Ray Workloads with KubeRay and Kueue in Kubernetes
32mAn Overview of CloudKitchens's Ray-Powered ML Platform
19mScaling Frontier AI with Ray on Kubernetes
28mRay on Public-Cloud k8s - experiments, lessons & best practices
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
31mNext-Generation Data Science Workflows Using Ray
32mRay in Production: Engineering Lessons from Spotify's Hendrix Framework
28mScaling Multimodal Data and Reinforcement Learning with Ray