These 10000 classes I never wrote
summary
Claim: use hylomorphisms and recursion schemes (matryoshka) to avoid 10,000 handwritten case classes. Tools: Spark, Kafka, Avro, Parquet, Play JSON validation and the matryoshka library. Team of five ships a single runtime‑configurable pipeline in six months that validates/transforms data from dozens of tables across hundreds of sources, encodes privacy by design in schemas, generates Avro/Parquet/Row converters, and actually ships on time and on budget.
AI-written summary · about
chapters (8)
- 0:00Start
- 7:26Recursion Schemes
- 11:30Pattern Filter
- 15:36Performance Penalty
- 16:31Algebra
- 27:13Validate Data
- 32:05Utilities Patterns
- 38:18Geogebra
related talks
44mRecursion Schemes Without the Barbed Wire (Part 1)
36mData pipelines engineering made simple with Scala
54mDo you really need Hibernate?
57mLessons in Building a System that Processes More than 70 Billion Events Daily
43mScaling Full-Stack Applications
52mData Over Code: A Short Hyperbole
54mDo you really need Hibernate
39mData-centric Metaprogramming