Interactive Machine Learning on Genomics Data with Spark Notebook
summary
Converts 1000 Genomes BAM/VCF raw files (≈152 GB) into ADAM schema stored as Parquet, creating ~10,000 partitions so the same dataset loads in about one minute instead of hours; shows ADAM + Parquet and Spark Notebook wired to Spark ML for interactive preprocessing, computes allele frequencies, runs k‑means that cleanly separates three populations, and calls H2O deep learning from Spark for larger models — everything runs end‑to‑end on a cluster and produces reproducible results.
AI-written summary · about
chapters (7)
- 0:00Introduction to talk
- 2:20Genomics data overview
- 7:23Schema for structured genomics
- 10:291000 Genomes Project data
- 13:43Interactive analysis with Spark Notebook
- 17:34Machine learning on genomics data
- 20:46Key takeaways and summary
related talks
46mScalable Genomics with ADAM
25mScaling Up Genomics with Spark
46mKeynote: Life Science Research at Scale
01h 48mBig Data made easy with a Spark (3.0)
44mData Science at Scale with Apache Spark
17mMachine Learning at the Speed of Light with Spark
42mHow Novel Compute Technology Transforms Life Science Research
35mScalable Data Science with FHIR