Sequence and Concurrent Processing for Huge Genomic Data
summary
Argues that Clojure plus concurrency matches bioinformatics needs: sequence-processing via lazy sequences and map/filter stays simple while pmap/controlled thread pools give practical speed on multicore. Uses libraries named Crozier rhodium and crew gem for FASTQ/BAM handling, measures on a 12-core box with 2 GB of 1000 Genomes data, sees good scaling up to a few threads. Ships a full product Clovis browser (analysis, DB, reporting, viz) with 80% Clojure/ClojureScript code, Java 12% and Python 7%, and uses the system in real projects.
AI-written summary · about
chapters (25)
- 0:00Start
- 0:54Who am I?
- 1:31Who are we?
- 2:16Outline
- 2:42Terminology
- 3:20Human Genome
- 6:42Bioinformatics Tools
- 7:26Why Using Clojure?
- 8:52Genomic Data is Character Sequence x
- 10:03Clojure and Sequence
- 10:34Higher-order Functions
- 11:56Rapid Feedbacks are Needed
- 12:33Lazy Sequence
- 13:25Alternative: clj-biosequence
- 14:20Is Clojure Slow?
- 15:05Concurrency
- 15:56pmap
- 16:43Reducers
- 18:02Claypoole
- 21:03Latest performance
- 22:01Simplicity
- 22:40That is Why Using Clojure
- 24:20Chrovis
- 31:0199% is Written in Clojure...?
- 32:12Conclusion
related talks
27mPowerful Data Access in Clojure
57mLessons in Building a System that Processes More than 70 Billion Events Daily
36mSimplifying ETL with Clojure and Datomic
49mWorking on Cancer
26mClojure vs Sepsis: path to real time enterprise data science
25mScaling Up Genomics with Spark
29mAnalysing and sharing genetic data with Python
43mScaling Full-Stack Applications