A Scalable GA4GH Server Implementation
summary
A GA4GH server implementation that claims to be a scalable reference uses Apache Spark and the ADAM genomics schemas as its back end, implements variant queries (about half the GA4GH spec) and exposes custom microservices and ML model estimation/prediction endpoints. Deployment comes directly from a Spark Notebook to run jobs and register services, with caching via a Tachyon-like layer and tested across 4 to 40 nodes; the prototype runs and indexes services for discovery.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:18What is GA4GH
- 3:43GA4GH Working Group
- 5:06What could we discover
- 5:42How can we correlate
- 6:25Scalable systems
- 7:48Schemas
- 9:20Strict Schema
- 10:23Scalable Server
- 12:10Spark Inside Atom
- 14:46Scalable Implementation
- 16:04Services Everywhere
- 17:00Genomics Catalog
- 19:08Deploy
related talks
02mPreview of Apache Spark 4.0 and GA of Spark Connect
07mCloud architecture for population-scale analyses of genomes and AI.
50mDesigning a scalable data platform based on Apache Spark/Iceberg
42mApache Spark on planet scale
34mApache AGE
24mLarge Scale Graph Computation with Apache Giraph
19mWhat's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0
52mSolving Analytical Problems using Apache Spark