THOSE 10000 CLASSES I NEVER WROTE
summary
Five developers ship a data-ingest pipeline in months that handles 10,000+ data structures across hundreds of sources by encoding formats (JSON, CSV, Avro, Parquet) as a single recursive schema and deriving converters and validators automatically. Implementation uses matryoshka recursion schemes (hylomorphisms, algebras, monadic algebras) plus a fingerprint registry to reuse named Avro records, JDL rules for validation, and a para for row building to emit Spark Rows and generic Avro records; the approach proves production‑ready.
AI-written summary · about
chapters (5)
related talks
31mOne Data Pipeline to Rule Them All
19mDataEngBytes 2023 - MEL-T1-04 - Michael Hyatt
58mHow to transform AVRO (IDL) data to multiple PARQUET files
39mComplex Data Transformations Made Easy
29mHigh-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale
38mEnabling Big Data and Machine Learning for the Masses...
57mOur DataMesh Journey: data architectures for microservices
49mTransforming Analytical Data with Event-Driven Architecture