Data-Shuffler: Bringing Order to The Galaxy (of Data Process)
summary
A Clojure-based ETL called Data-Shuffler drives Apache Spark with declarative SQL transforms stored in a giant 10,000-line Clojure data structure, validated by clojure.spec before execution. It runs nightly, supports producing Hive and Postgres outputs, and lets teams programmatically re-point I/O for high‑fidelity tests against real sinks. Results: ~30 contributors, roughly 3× more datasets per capita, CI cut from ~12 hours to ~45 minutes, and much fewer runtime surprises.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:16About me
- 0:44JSTOR
- 1:58Data Platform
- 2:25Data Pipeline
- 3:47ETL System
- 4:51Data Explosion
- 5:35The Model
- 6:57The Goal
- 8:17The Right Way
- 8:30The Declarative Way
- 9:35Apache Spark
- 10:52Spark
- 11:35Closure Spec
- 12:26Sidestep Building UI
- 13:36Data Structure
- 16:17Spec Structure
- 17:57Validation
- 20:16Polymorphism
- 22:09The blurred line
- 22:53Testing is critical
- 24:11A real sequel query
- 24:31Testing is hard
- 25:51Our approach
- 27:48End result
- 28:30Workflow
- 29:39Productivity
- 30:51Conclusion
related talks
31mOne Data Pipeline to Rule Them All
39mComplex Data Transformations Made Easy
35mPowderkeg: teaching Clojure to Spark
36mIn-memory data pipeline and warehouse at scale
42mBuilding Robust Streaming Data Pipelines with Apache Spark
31mSpark challenges
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse
38mEnabling Big Data and Machine Learning for the Masses...