Keynote: Life Science Research at Scale
summary
Shows how VariantSpark random forest runs on Spark to analyze 22,000 genomes (≈80 million variant columns, a 1.7 trillion‑cell matrix) to surface ALS-associated variant sets rather than single SNP hits, and benchmarks scaling and accuracy against other tools. Describes a serverless genomic search GT‑Scan implemented on AWS (also on Alibaba), an infrastructure‑as‑code CloudFormation workflow, and a measured 80% runtime cut after replacing two slow lambdas with an ML classifier. Advocates hypothesis‑driven DevOps and Epsagon for distributed monitoring.
AI-written summary · about
chapters (9)
- 0:00Start
- 3:44Genomics affects looks, disease risk, and behavior
- 6:52Finding disease gene(s)
- 8:59Machine learning on 1.7 Trillion datapoints
- 16:26Typical workflow cloud pipeline pattern
- 26:03The search engine for the genome
- 30:12Differences Aliyun vs AWS
- 33:25Usinghypothesis-driven architecture
- 40:40Three things to remember
related talks
45mDigital Disruption in health & medical research
42mHow Novel Compute Technology Transforms Life Science Research
50mCancer Genomics - a biologist and a developer
25mScaling Up Genomics with Spark
21mInteractive Machine Learning on Genomics Data with Spark Notebook
18mKeynote: Machine Learning & Apache Spark: A Dynamic Duo
44mData Science at Scale with Apache Spark
46mScalable Genomics with ADAM