Achieving Scalability and Interactivity with Ray Serve
summary
Ikigai Labs builds Eki, a pipeline platform that lets users inject arbitrary Python steps and needs both massive scale and sub-second peeking. Ray and Ray Serve power the solution: each custom step runs as a deployment with conda per deployment (actor options embed envs) so dependency conflicts vanish, and Serve exposes HTTP endpoints to avoid the 7–10s Ray job submission overhead and deliver sampled rows under a second. An in-house Ray Serve Manager creates/upgrades deployments, manages envs, tunes replicas and deployment versioning; Ray Client is next to remove a central bottleneck.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:54The Ikigai Platform: Al-Charged Spreadsheets
- 2:23Outline
- 2:49Mission-Critical Data Pipelining
- 3:49Highly Interactive Data Pipelining
- 4:13Instantly Browsable Data Pipelining
- 5:37Earning Scalability with Minimal Code Change
- 6:39Introducing Ray Serve
- 7:41Challenge: Conflicting Python Libraries
- 8:38Defining Custom Python Scripts with Ray Serve
- 9:04Dependency Management with Ray Serve
- 9:29Challenge: Task Overhead for 'Peek
- 10:24Dynamic Function Invocation with Ray Serve
- 14:42Challenge: Skewed Traffic on Serve Instances 8888 888
- 15:27Scaling a Specific Serve Instance Individually
- 15:53Challenge: Deployment Race Condition
- 16:28Explicit Deployment Versioning
- 18:26Even More Flexibility with Ray Client
- 19:28Data Pipelining without Task Submission
- 20:14Ikigai with Ray Serve and Client
related talks
23mIntroducing Ray Serve: Scalable and Programmable ML Serving Framework
24mBuilding Complex Data Analytics Pipelines with Ray
15mRay: A System for Distributed Applications
31mNext-Generation Data Science Workflows Using Ray
31mKlaviyo's Journey to Robust Model Serving with Ray Serve
26mRay: A System for High-performance, Distributed Python Applications
22mRay, a Unified Distributed Framework for the Modern AI Stack
32mAn Overview of CloudKitchens's Ray-Powered ML Platform