Building a Virtual Biology Platform on Ray
summary
Recursion runs a virtual biology platform on Ray and Anyscale over a 60 PB dataset with 40 PB proprietary, claiming to cut cloud spend while improving reliability. They replace brittle Kubernetes and Prefect jobs with Ray array tasks that do concurrent download + inferencing and retry at task granularity, add plate-level GPU cost metrics, and route jobs across GCP and on‑prem BioHive GPUs (H100/A100 pool) using a priority-aware router called cake. Migration takes about a year and yields better observability, portability, and cost control.
AI-written summary · about
chapters (15)
- 0:00Introduction to virtual biology
- 2:00Recursion’s platform and mission
- 5:16Four-step drug discovery loop
- 6:22Phenomics pipeline challenges
- 10:40Requirements for new platform
- 12:51Why Ray and Anyscale
- 15:40Anyscale as infrastructure layer
- 16:01Phenomics pipeline today
- 19:12Portability across clusters
- 20:15Compute routing with CAKE
- 21:19Transcriptomics on Ray
- 22:47Predictive chemistry workloads
- 24:15Virtual cell simulations
- 27:04Building a unified platform
- 28:07Q&A session
related talks
24mBuilding Complex Data Analytics Pipelines with Ray
26mData Processing on Ray
26mUsing Ray On Large-scale Applications at Ant Group
30mHow to Scale Unstructured Data Processing with Ray Data
29mAdopting Ray @ LinkedIn
31mNext-Generation Data Science Workflows Using Ray
21mLeveraging Ray to Enhance Machine Learning Models in Finance
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes