Clojure Dispatches From a National Research Lab
summary
At Sandia a pipeline takes heterogeneous research files as Clojure maps, validates them with clojure.spec multi-spec dispatch, encodes with Transit and accepts them over HTTP Kit/Compojure-API. Kafka fans out messages to pluggable Kafka-backed writers managed by Component; writers persist into Datomic (via Munger) so historical state and provenance are available. GraphQL-clj provides a language-agnostic query layer. Result: working ingestion, quick researcher access, smaller codebase, schema evolution and auditability without custom parsers for every format.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 0:17Overview
- 0:54Outline
- 1:18About Sandia
- 2:20Background
- 3:11Marc Bastian
- 4:50Sandiagov
- 5:13Data
- 6:29Data in Practice
- 7:25Challenges
- 9:04Our Approach
- 11:52Spec
- 13:49MultiSpec
- 14:44Transit
- 15:26HTTP GET
- 16:08The Front Door
- 16:26Comprehensible API
- 17:48Kafka
- 19:33Component
- 20:15Component Map
- 21:33Mongo
- 22:08Diatomic
- 24:03GraphQL
- 27:02A Critical Success
- 27:37Smaller Code Base
- 28:14Composable Parts
- 28:42Interop
- 29:49Closure
- 30:31Speck
- 31:17Moving Forward
- 31:58Solving the Problem
- 32:36Conclusion
related talks
43mDoing Data Science with Clojure
38mBuilding a Legal Data Service with Clojure
37mCascalog 2- Streaming MapReduce in Clojure
38mScientific Clojure, a bird's eye view
37mClojure, a Sweetspot for Analytics
46mClojure Scaling the Event Stream
22mPanel with Alexander Oloo and Johanna Antonelli
24mSpeculative Development