Processing medical images at scale on the cloud
summary
Whole Slide Images (WSIs) are tens of gigabytes each so the project shows how to avoid copying whole files, read only the tiles needed, and feed patches into GPU training at scale. It uses OpenSlide or a drop‑in fsspec-backed reader, Azure blobfuse2 or fsspec streams to read only relevant bytes, converts tiles to patches for PyTorch, and runs distributed preprocessing with Ray so a small GPU head plus many cheap CPU workers reaches the required ~90x parallelism. Ray datasets map lazy distributed patch reads
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:26What we will cover in this talk
- 2:32Oncology and imaging basics
- 4:21Whole slide images challenges
- 7:10Cloud processing benefits
- 8:14Reading whole slide files
- 10:47Cloud file reading challenges
- 11:30Solution: download entirely
- 11:52Solution: FUSE for cloud
- 13:18FUSE limitations
- 14:00Solution: TIFFSlide library
- 15:47Dataset preparation
- 16:29Efficient patching
- 17:29Data loading bottlenecks
- 18:36Parallelism and distribution
- 19:18Simplify with Ray
- 21:02Ray on Kubernetes
- 22:50Distributed training with Ray
- 24:15Local development
- 24:37Summary
related talks
29mBuilding Self-Serve, Multimodal Medical Image Pipelines with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
30mHow to Scale Unstructured Data Processing with Ray Data
34mDeep Learning at Scale
24mTiling & Parallel Processing of Large Images
33mOvercoming Distributed ML Challenges with Ray Train
29mRay Data on GPU - Breaking Preprocessing Bottleneck for Model Training
35mOperating a Production-Scale Physical AI Data Platform with Ray