Machine Learning and Microbiology in the Cloud Through Ray
summary
A cloud pipeline is built that uses JupyterLab to author code, Airflow to DAG and schedule, Ray to execute distributed jobs, Kubernetes/Docker for images, Slurm for legacy handoffs, and AWS for compute; the team packages genome prediction as a three‑step pandas→preprocess→train flow and deploys it from notebook to cluster. They report a 10:1 skill gap between cluster-savvy and novice data scientists and name the system Falconer, inspired by the infinite laptop idea, and describe it as working in production.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:39Drift Bio Technologies
- 4:15Genetic Sequencing
- 5:07Bioinformatics Challenges
- 6:29Cloud Engineering Challenges
- 13:11Example Systems
- 18:07Falconer
related talks
32mAn Overview of CloudKitchens's Ray-Powered ML Platform
31mNext-Generation Data Science Workflows Using Ray
24mBuilding Complex Data Analytics Pipelines with Ray
21mLeveraging Ray to Enhance Machine Learning Models in Finance
35mOperating a Production-Scale Physical AI Data Platform with Ray
54mScaling Machine Learning Pipelines in Cloud
24mRay: A System for High-performance, Distributed ML Applications
31mBuilding a Virtual Biology Platform on Ray