Powering Open Data Hub with Ray
summary
Integrates Ray 2.0 client-server into Open Data Hub by adding Kubernetes configmap-based single-user profile and service-template entries that spawn on-demand Ray clusters (head + workers) and leave Jupyter pods separate. Deployment uses the Ray operator to autoscale clusters, tested on Massachusetts Open Cloud with a cap of one head plus five workers (1 CPU, 1 GB each). Demo runs a Ray-based Monte Carlo Pi, returns ~3.14 with parallel tasks, and the setup works; configs and repo are public.
AI-written summary · about
chapters (13)
- 0:00Introduction to talk
- 0:22Why Jupyter with Ray
- 2:50Ray 2.0 architecture change
- 4:14Open Data Hub context
- 8:04Architecture analogy with Spark
- 11:16Demo deployment overview
- 12:20Ray Pi Monte Carlo demo
- 18:59Benefits of Jupyter and Ray
- 19:20Massachusetts Open Cloud setup
- 20:46Operate First collaboration
- 21:49IBM collaboration details
- 23:13Future roadmap plans
- 24:14Conclusion and resources
related talks
21mPowering Open Data Hub with Ray
31mNext-Generation Data Science Workflows Using Ray
29mAdopting Ray @ LinkedIn
28mRay on Public-Cloud k8s - experiments, lessons & best practices
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
32mAn Overview of CloudKitchens's Ray-Powered ML Platform
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
22mRay, a Unified Distributed Framework for the Modern AI Stack