One Data Pipeline to Rule Them All
summary
Built a Kafka-backed event pipeline to replace N^2 point-to-point ETLs, moving all records through Apache Kafka with Avro schemas validated at produce time and a central schema registry. HTTP ingress and language libraries enforce schema versions; archive via copycat to S3 and a Spark job called copydog deduplicates, orders and materializes archives. Redshift for analytics, Spark for batch and streaming, consumer offset tracking and lightweight reconciliation ensure correctness; supports a 400-person R&D org and enables safe replays and schema evolution.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:55Show of Hands
- 4:38What Went Wrong
- 6:17The Solution
- 11:45Schemas
- 14:34Enforce
- 17:20Systems Libraries
- 20:15Systems
- 27:15Data Accuracy
- 28:15Recap
related talks
43mBuilding Scalable and Extendable Data Pipeline
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
45mData pipelines from zero to solid
42mBuilding Robust Streaming Data Pipelines with Apache Spark
42mBuilding a Data Pipeline with Clojure and Kafka
34mTales of Data Architecture Evolution
39mData Pipeline Evolution
42mWeb Development To Big Data. A Journey