Ray Summit 2024
2024-09-30..2024-10-02
San Francisco, United States
[Ray Summit 2020, Ray Summit 2021, Ray Summit 2024, Ray Summit 2026]
as the organisers put it
Powered by Ray, Anyscale empowers AI builders to run and scale all ML and AI workloads on any cloud and on-prem.
| title | author | length |
|---|---|---|
| Meta's Roadmap for Full Stack AI | Joe Spisak | 38m |
Llama drives 400M+ downloads and over 65,000 derivatives and Meta reports models trained on large token counts (the 4–5B class sees ~15T tokens). New releases… | ||
| The Evolution of Multi-GPU Inference in vLLM | SangBin Cho, Murali Andoorveedu | 31m |
vLLM shards 405B-class models across GPUs using tensor + pipeline parallelism, rearchitects the control plane to use Ray compiled graph with GPU RDMA and Niel… | ||
| The State of vLLM | Kuntai Du, Zhuohan Li | 35m |
vLLM is an open‑source LLM inference and serving engine (origin Aug 2022, open‑sourced June 2023) that now powers production demos and cloud integrations and… | ||
| Scaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve | Neelay Shah, Ryan McCormick, Akshay Malik | 32m |
Integration embeds the Triton Python API inside Ray Serve deployments so Python decorators launch Triton-managed models (PyTorch/ONNX/TensorRT) with zero‑copy… | ||
| Building LLaMA: Meta's Director of GenAI | Sergey Edunov | 31m |
Llama evolution moves from a 1.4 trillion tokens research-only pretrain (2k context) to instruction-tuned Llama 2 at 2 trillion tokens (4k context) and a Llama… | ||
| Optimizing vLLM Performance through Quantization | Michael Goin, Robert Shaw | 38m |
vLLM adds practical quantization paths — activation quantization (FP8/INT8) and weight-only (W4/W8) — plus conversion kernels Marlin and Machete and a repo of… | ||
| How NVIDIA is Advancing Video Curation with Generative AI | Niket Agarwal | 34m |
NVIDIA describes an end-to-end video curation service running on DGX Cloud and orchestrated with Ray Data, processing >100 PB of raw video down to tens of PB… | ||
| How Rad AI is Revolutionizing Radiology with Ray | Ken Kao, Priunsh Syen | 31m |
Rad AI automates radiology impressions with Radi Omni, generating ~100,000 impressions/day for ~4,000 daily radiologists and claiming up to 90% fewer words and… | ||
| The Future of AI Assistants | Jerry Liu | 29m |
Argues that naive RAG fails for complex enterprise data and shows concrete fixes: an LLM-native parser (Lama Parse) that extracts tables, charts and captions… | ||
| Ray at Scale: Apple's Approach to Elastic GPU Management | Weiwei Yang, Abin Shahab | 31m |
Apple runs Ray on Kubernetes with Apache YuniKorn queues and enforces per-queue guaranteed vs max GPU quota so teams get baseline GPUs while idle capacity is… | ||
| ByteDance's Platform for Reinforcement Learning from Human Feedback | Haibin Lin, Chi Zhang, Liguang Xie | 32m |
ByteDance describes a production RLHF platform that runs LLM trajectory generation and training on Ray while keeping model-parallel LLM compute inside SPMD… | ||
| Anyscale's Ray Data: Revolutionizing Batch Inference | Richard Liaw, Scott Lee | 32m |
Ray Data plus Anyscale runtime implements a streaming executor with actor-based GPU workers and an optimizer, claiming Ray LM batch as a proprietary library… | ||
| Pinterest's ML Evolution: Distributed Training with Ray | Saurabh Vishwas Joshi, Jiun-Yu Lee | 27m |
Pinterest uses Ray and Ray Data to build a distributed data loader that moves CPU-heavy file IO and preprocessing off GPU trainers, implements batching as Ray… | ||
| Transforming Multimodal Data Management with LanceDB-Ray | Lei Xu, Change She | 31m |
LanceDB presents a columnar, object-store native format that stores images, video and embeddings inline and adds columns without rewriting data, enabling… | ||
| Netflix's Ray Platform: From Deep Learning to GenAI | Pablo Delgado | 38m |
Netflix runs daily personalized-model training on a 24‑hour cadence and scales to pools like 800‑GPU P4 clusters plus CPU pools and FSx for high throughput. A… | ||
| How Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray | Xiaohong Dong, Zhibei Ma, Liguang Xie | 35m |
ByteDance builds a Ray-based pipeline (Ray Core, Ray Data, Ray Serve) to process petabyte-scale audio/video for generation models, using FFMPEG for… | ||
| Airbnb's LLM Evolution: Fine-Tuning with Ray | Shaowei Su, Mia Zhao | 26m |
Airbnb migrates its model layer onto Ray and builds a one-job-per-cluster pipeline that combines Llama Factory for unified open‑source models, Kueue for… | ||
| Databricks' vLLM Optimization for Cost-Effective LLM Inference | Megha Agarwal | 28m |
Databricks reports concrete vLLM fixes: profiling shows GPU idle time climbs from ~2.6% at batch 1 to ~25% at large batches for 70B models, and release 0.6.0+… | ||
| How Roblox Scaled Machine Learning by Leveraging Ray for Efficient Batch Inference | Steve Han, Wei Zeng, Yiqing Wang | 29m |
Roblox integrates Ray (Ray Data, Ray Jobs) with KServe, Triton, Kubernetes and vLLM to run daily batch inference jobs that process up to 10+ billion tokens per… | ||
| Building Scalable AI Agents with Ray | Guyang Song, Qixiang Chen | 29m |
Ragent is a Ray-based framework that distributes agents as actors and tasks, uses Ray Core for scheduling and Ray Serve for real-time endpoints, and runs… | ||
| Advanced Evaluation Techniques for AI Agents | Jason Lopatecki | 36m |
Claims a practical evaluation stack for LLM agents: a simple LM router -> skill -> router architecture (90% of deployments), function calling as the core… | ||
| Coinbase's ML Training Evolution: From Sagemaker to Ray | Wenyue Liu | 35m |
Coinbase replaces Sagemaker with Ray and rebuilds training as small wheel/zip deployments plus Ray Data and a long‑running cluster, cutting PR→run iteration… | ||
| Spotify's Approach to Distributed LLM Training with Ray on GKE | Abdullah Mobeen, Brandon Royal | 31m |
Spotify trains LLMs larger than 70B parameters using Ray on GKE with H100 GPUs, compact placement and NCCL Fast Socket to cut inter‑GPU latency, and uses… | ||
| Scaling Ray Train to 10K Kubernetes Nodes on GKE | Andrew Sy Kim, Saikat Roychowdhury | 35m |
Google runs Ray Data and Ray Train across a 10,000-node GKE cluster by sharding Landsat‑8 (≈3.3 PB) into about 22,000 Ray jobs, using KubeRay plus a… | ||
| How Samsara Built an Agentic Framework for GenAI Applications | Yangyong Zhang, Brian Westphal, Sven Eberhardt | 32m |
Samsara builds an agentic, RAG‑centric system that ships a knowledge‑base chatbot in days using LlamaIndex and a modular agent API, runs parallel expert agents… | ||
| How to Scale Unstructured Data Processing with Ray Data | Hao Chen, Praveen Gorthy | 30m |
Ray Data on Anyscale claims to solve unstructured workloads by combining a streaming batch model with adaptive scheduling so GPU and CPU stages run… | ||
| Reddit's ML Evolution: Scaling with Ray and KubeRay | Cedric Blondeau, Garrett Hoffman | 35m |
Reddit moves its training and serving stack onto Ray and KubeRay, building Ray workspaces with a persistent Jupyter head node, a Git-tagged Docker release… | ||
| Evolving Ray Core: Scalability, Reliability, and Compiled Graphs | Stephanie Wang, Rui Qiao | 37m |
Weekly releases and structured logging reduce release latency and improve observability, streaming generators let callers consume object refs while tasks still… | ||
| Spotify Harnesses Ray for Next-Gen AI Infrastructure | Abdullah Mobeen, Grace Glenn | 30m |
Spotify runs Ray on GKE and claims support for LLM training beyond 70B parameters, growing Hendrix from 80 to 120 teams and from ~100 to ~1,400 Ray… | ||
| Optimizing Large-Scale Model Training with Ray Compiled Graphs | Yunxuan Xaio, Amjad Almahairi | 28m |
Ray Compiled Graphs create a static Ray-task graph with native GPU‑GPU NCCL communication and precompiled execution paths, letting different processes run… | ||
| Roblox's Journey to Supporting Multimodality on vLLM | Roger Wang | 30m |
Roblox integrates multimodal inputs (image, audio, video) into vLLM and ships an end-to-end pipeline that is OpenAI Vision API compatible. Engineering delivers… | ||
| Faster Model Serving with Ray and Anyscale | Edward Oakes, Akshay Malik | 31m |
Ray Serve provides Python-native deployments with fractional GPUs, model multiplexing and integrations like vLLM, TensorRT LLM and NVIDIA Triton, enabling… | ||
| Uber's GenAI Leap: Batch Predictions Using Ray and vLLM | Anant Vyas, Baojun Liu, Bo Ling | 33m |
Uber integrates Ray + vLLM on Michelangelo to run large GenAI batch prediction at scale, replacing Spark for GPU-heavy workloads. The system handles millions… | ||
| How The AI Institute is Revolutionizing Robotics ML Training | Ahmet Salih Gundogdu, Erica Lin | 35m |
Boston Dynamics AI Institute ships an end-to-end Python platform called RobotTensors platform that glues Ray Data, Ray Train, Ray Tune and RayClusters to GKE… | ||
| How Zoox Accelerated Autonomous Driving with Ray | Rong Zhou, Eli Stevens | 22m |
Zoox describes how Ray (Train, Data, Serve) plugs into an existing SLURM-backed on‑prem cluster and AWS EKS to power both multi‑node PyTorch Lightning training… | ||
| Dynamic Scheduling for Large Language Model Serving | Hanyu Zhao | 32m |
Llumnix is a dynamic scheduler built on vLLM and Ray that implements near-zero-downtime live migration of KV caches to continuously reschedule requests across… | ||
| Ray @ eBay: Pioneering a Next-Gen AI Platform | Yucai Yu | 32m |
eBay integrates Ray across batch inference, near‑real‑time feature pipelines, distributed training and real‑time serving, using Ray Data, Triton, fractional… | ||
| Shopify's Ray-Powered Approach to Multimodal AI in E-commerce | Niklas Itanen, Kshetrajna Raghavan | 31m |
Shopify builds a production pipeline that fine-tunes vision LLMs to map unstructured titles, descriptions and images into a 12,000-node product taxonomy with… | ||
| Streamlining AI Workflows with Apache Airflow and Ray | Venkata Jagannath, Marwan Sarieddine | 21m |
Shows a runnable Airflow + Ray + Anyscale pipeline that continuously ingests Twitch and Discord feedback, stores it to S3, then Mistral 7B fine-tuned with LoRA… | ||
| Exploring Hinge's ML Platform Evolution with Ray | Dev Goyal, Jonathan Jin | 23m |
Hinge migrates from EMR Spark to Ray on Kubernetes to shorten time‑to‑production, running Ray clusters via Argo CD for GitOps with GitHub Actions building… | ||
| Building Intelligent AI Infrastructure with O.XYZ's ORI | Haider Khan, Zac Liu, Heather con | 22m |
ORI routes decomposed user queries as a chain of atomic queries, sends each subquery to the top-performing open‑source LM (matched by a benchmark matching… | ||
| How Uber Optimize Marketplaces with Ray | Peng Zhang, Kaichen Wei, Matt Walker | 32m |
Claim: Ray delivers up to 47.8x speedup across Uber’s marketplace workflows by parallelizing per-city per-week ADMM subproblems. Implementation: a hybrid… | ||
| Intelligent Data Classification with Ray and vLLM at Apple | Praveen Kanamarlapudi | 25m |
Apple shows a working pipeline that fine tunes LLMs with Ray and vLLM and Ray for low latency online and high throughput batch metadata generation, uses… | ||
| Uber's Journey in Building Multi-Modal Foundation Models | Ivan Zhou, Sabarish Sivanath | 35m |
Uber builds an in‑house encoder‑decoder multimodal foundation model trained on about 50 million documents across 120 document types to automate driver… | ||
| Overcoming Distributed ML Challenges with Ray Train | Matthew Deng | 33m |
Ray Train is a Python library that sets up distributed training from one line of scaling config, launches workers, and configures framework-specific… | ||
| Optimizing vLLM for Intel CPUs and XPUs | Ding Ke, Yuan Zhou | 30m |
vLLM is integrated with Intel’s IPEX backend to run on Xeon CPUs and Intel GPUs/XPUs with zero code change for users; integration is op-level (KV cache… | ||
| Anyscale's Unified Platform for LLM Development and Deployment | Kourosh Hakhamaneshi | 28m |
Anyscale offers a unified stack that actually deploys and scales LLMs: raym provides an OpenAI‑compatible serving API with autoscaling, fast model… | ||
| Chatbot Arena: UC Berkeley's Open AI Evaluation Platform | Wei-Lin Chiang | 32m |
An open platform for human-preference LLM evaluation collects millions of pairwise votes and runs live side-by-side comparisons, reporting 1.7 million user… | ||
| An Overview of CloudKitchens's Ray-Powered ML Platform | Santosh Jha, Ammar Alrashed | 32m |
City Storage Systems moves its ML stack onto Kubernetes and Ray, replacing legacy Trino/Flink pieces and proving Ray can host ETL, training, and serving with… | ||
| Managing Large-Scale Ray Deployments: Cloud, On-Prem, and Kubernetes | Bruce Zhang, Shomil Jain | 31m |
Anyscale describes a platform that runs over 10,000 Ray clusters and routinely manages millions of cores and GPUs and demonstrates a single Ray cluster scaled… | ||
| How IBM Research Achieved vLLM Platform Portability with Triton Autotuning | Burkhard Ringlein | 31m |
IBM Research replaces hand‑tuned CUDA kernels in vLLM with OpenAI Triton kernels plus an autotuning cache called Triton djaVu, which records best block‑size… | ||
| Customizable Cluster Metrics Made Easy with Raydar and Anyscale | Todd Gaugler | 32m |
Raydar is an open source pip package that you can pip install into a Ray runtime to capture task metadata outside the Ray GCS and tracks 250,000+ tasks live… | ||
| Accelerated LLM Inference with Anyscale | Philipp Moritz, Cody Yu | 30m |
Anyscale details concrete LLM inference gains: contributions to vLLM add FP8 support (benchmarks show ~1.27x TTF and 1.5x token latency on H100), chunked… | ||
| Scaling ML Pipelines for Autonomous Driving with Ray | Yi Wang | 29m |
Motional reworks end-to-end ML with Ray, replacing weekly reloads with a pipeline that reduced data delivery latency from weeks to hours by streaming and Ray… | ||
| Using Promptable Image Analysis for Clinical Diagnostics | Robert Berke | 33m |
Kaiko.ai builds a multimodal clinical assistant called Kao Insight that fuses ViT image encoders with vLLM-style language models, deploys training on Ray Train… | ||
| Klaviyo's Journey to Robust Model Serving with Ray Serve | Smith Carey | 31m |
Klaviyo describes a production DART (DAtascience RunTime) model-serving platform built on Ray Serve and KubeRay, running two identical Ray clusters behind… | ||
| Canva's AI Leap: Scaling with Ray and Anyscale | Amar Čivgin, Kelly Ke | 26m |
Canva reworks its training platform with Ray and Anyscale, cutting typical turnaround by 25× and migrating in about six months to ~30 clouds running ~2,000… | ||
| How Vivix Scales Video Ad Classification with Ray | Thierry Steenberghs, James Kim | 31m |
Company scales ad classification from 30k to a 6 million label target by piping video into Event Hubs, Spark Structured Streaming on Databricks and Ray for… | ||
| Handshake's Approach to Content Tagging with vLLM and Anyscale | Kyle Gallatin, Nabil Ahmed | 24m |
Handshake implements vLLM on Anyscale and Ray to run in‑house LLM inference, keeping PII internal, and pairs that with Airflow + BigQuery batch jobs for… | ||
| IBM's Approach to Building a Cloud-Native AI Platform | Carlos Costa | 33m |
IBM and Red Hat describe an open source cloud‑native stack that combines KubeRay & Ray, PyTorch (FSDP for distributed training), Triton CUDA‑free kernels, and… | ||
| How Genmo Scales Video Generation with | Paras Jain | 33m |
Genmo claims production text-to-video at scale: over 1.5 million users, thousands of GPUs across 20+ data centers, and a training supercomputer ranked top‑50… | ||
| Generative AI + Proprietary Data | Raluca Ada Popa | 28m |
Shows a practical stack that keeps prompts, RAG DBs and model weights encrypted while running by combining memory encryption and remote attestation on enclave… | ||
| Enabling Multi-User Clusters with | Donny Greenberg, Matt Kandler | 30m |
Runhouse adds a Python-first layer that orchestrates work across remote Ray clusters (multi-region and multi-cloud) via HTTP so a single script can call… | ||
| How Workday Fine-Tunes LLMs at Scale | Trevor DiMartino | 31m |
Workday builds a fine-tuning platform that trains tenant-specific LoRA adapters on isolated customer data, using DeepSpeed to stitch many single‑GPU instances… | ||
| Training AI for Space: | Alex Post | 21m |
Claims a production pipeline that runs multi-agent, multi-mission RL training on a serialized, configurable RL Gym that swaps simulation engines via a simple… | ||
| Optimizing GenAI with Intel Gaudi Accelerators on | Carson Wang, Chendi Xue, chany sh | 28m |
Ray + Intel Gaudi integration delivers concrete results: native Ray Core/Train/Serve and a VM Gaudi project are available in container images (software v1.7… | ||
| Streamlining Data Access for with | Haoyuan Li, Bin Fan | 32m |
Alluxio integrates with Ray as a distributed caching layer sitting between GPUs and object stores (S3/GCS/Azure), claiming petabyte scale and billions of files… | ||
| Governance Suite for Scalable AI Platforms | Dominic Catalano, Allen Yin | 25m |
Anyscale’s Governance Suite claims to give platforms concrete controls for security, cost, observability and automation: organization/cloud/project/resource… | ||
| Revolutionizing ETL for RAG with | Benjamin Sowell, Ben saell | 32m |
Sycamore is an open-source ETL system built on Ray that turns messy PDFs into a tree-structured Sycamore DocSet for scalable ETL and analytics, using a custom… | ||
| nothing matches | ||