Cascade: Write-Once Run-Anywhere Inference on Ray
summary
Cascade compiles DSL-defined DAGs into Ray Serve, Ray Data or Spark runtimes so one logical ML pipeline runs unchanged for HTTP real‑time serving and S3/parquet batch backfills. It turns stages into Ray primitives, propagates tracing for 42s end-to-end trace visibility (27s queued waiting), and separates CPU preproc from GPU inference to raise average GPU use from 19% to 55% — about 2.9x GPU utilization — across millions of dashcams and 25 trillion annual datapoints.
AI-written summary · about
chapters (13)
- 0:00Introduction to Samsara
- 2:36Physical AI challenges
- 6:31Introducing Cascade
- 8:58Cascade architecture overview
- 11:06Logical plan in code
- 12:54Runtime compilation targets
- 13:57Resource allocation per stage
- 14:40Pipeline patterns support
- 15:45Configuration file example
- 16:26Observability and metrics
- 18:16Independent scaling benefits
- 20:06Key takeaways summary
- 21:32Closing and team recruitment
related talks
17mScaling and Unifying SciKit Learn and Spark Pipelines using Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
03mRay: A Distributed Execution Framework for Emerging AI Applications
32mAnyscale's Ray Data: Revolutionizing Batch Inference
26mRayDP: Build Large-scale End-to-end Data Analytics and AI Pipelines Using Spark and Ray
29mScaling ML Pipelines for Autonomous Driving with Ray
24mBuilding Complex Data Analytics Pipelines with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery