Simplifying ETL with Clojure and Datomic
summary
Converts the MusicBrainz dataset from Postgres into Datomic using Clojure, Datomic client API, spec, transducers and core.async and ends up with a restartable, batchable pipeline that actually imports ~100 million datoms. Code totals: ~150 lines for the job, ~50 lines of specs, ~100 lines of lookup tables, plus small generic helpers; transducers are under 50 lines. Uses reified transactions with batch IDs for provenance and transducers + core.async to localize errors and enable parallel, resumable loads; it succeeds and will be open sourced.
AI-written summary · about
chapters (20)
- 0:00Intro
- 2:14extract, transform, load
- 3:43functional pipeline
- 5:26systemic generality
- 6:06information
- 7:58correct / agile / robust
- 9:19input entity specs
- 11:25conformance check
- 12:08less obvious advantages
- 12:46transducers
- 14:26emergent simplicity
- 18:06place bias
- 19:40decision bias
- 21:08strong namespaces
- 21:32spec cohabitation
- 22:57reified transactions
- 23:19transaction attributes
- 25:21example datoms
- 25:42universal schema benefits
- 32:29lean and simple
related talks
43mDatalog all the way down
40mThe power of Datomic database
32mClojure is a Foreign Country: Combining Datomic with Scala
33m¿Where's My Data?
42mYet & Datomic Immutable Facts Mutated Our Stack
36mImmutable Data Science with Datomic, Spark and Kafka
37m11 insights after 11 years with the functional database Datomic
51mDatomic