Cloud Science
summary
GRAIL builds and runs Reflow, a vertically integrated language+runtime that transparently parallelizes Dockerized bioinformatics across EC2 (including spot instances), caches every result in S3, and performs fully incremental, lazy evaluation. The system is ~30,000 lines of Go, uses Docker images and S3/EC2, and routinely scales to 20,000 cores for noise-model training. Reflow memoizes outputs, restarts failed work cheaply, reduces repeated alignment costs, and is already production-used at GRAIL with measurable time and cost savings.
AI-written summary · about
chapters (9)
- 0:00Introduction to Grail and cancer detection
- 4:20Limitations of traditional HPC bioinformatics
- 7:59Introducing Reflow system design
- 13:12Reflow hello world demo
- 15:01Code walkthrough align module
- 22:04Incremental computation benefits
- 27:39Reflow runtime simplicity and robustness
- 30:54Reflow in production at Grail
- 32:19Q&A session
related talks
07mCloud architecture for population-scale analyses of genomes and AI.
36mAn almost perfect match: cloud native and scientific computing
31mBuilding a Virtual Biology Platform on Ray
21mMachine Learning and Microbiology in the Cloud Through Ray
27mVisualizing Cloud Systems
34mScaling Scikit Learn
06mReproducible Genome-Wide Analysis With Cloud Computing
23mGenomics and AI on Cloud Foundry