Scaling Up Genomics with Spark
summary
Genomics is framed as a data and compute problem: cost per genome falls to about a thousand dollars, and variant calling and recalibration improve when many genomes are processed together. The concrete stack uses HDFS/YARN, ADAM on Spark and modern columnar/record formats like Avro/Parquet formats to replace brittle BAM/VCF files, making joins and spatial queries trivial. Benchmarks show near linear scaling (4x machines -> 4x throughput) and pipelines shrink from miles of ad hoc code to compact Spark jobs.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:02Genomics as data problem
- 4:32Raw DNA to annotated genome
- 8:27Cost reduction in sequencing
- 10:58Bioinformatics pipeline challenges
- 13:30Legacy data format problems
- 15:59Scaling for population genomics
- 17:48Big data tools for genomics
- 19:54Modern genomics pipeline frameworks
- 21:18Spark benefits for genomics
- 22:40Contributing to genomics projects
- 23:38Q&A
related talks
46mScalable Genomics with ADAM
21mInteractive Machine Learning on Genomics Data with Spark Notebook
46mKeynote: Life Science Research at Scale
50mCancer Genomics - a biologist and a developer
42mHow Novel Compute Technology Transforms Life Science Research
44mData Science at Scale with Apache Spark
45mDigital Disruption in health & medical research
40mScaling with Apache Spark (or a lesson in unintended consequences)