The Changing Face of ETL: Event-Driven Architectures for Data Engineers
summary
Argues that modeling everything as events and persisting them in an event streaming platform (Apache Kafka) replaces brittle point-to-point ETL; demonstrates immutable commit log, per-topic partitions (example: 10 partitions for parallel consumers, preserving ordering per key) and retention policies (time-based or keep for 10 years or use compacted topics for latest key). Shows Kafka Connect, Kafka Streams and ksql for enrichment/cleansing, streaming to Elasticsearch and a data lake, and a hands-on demo that successfully replayed and processed events end to end.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:12The Changing Face of ETL
- 2:54It used to be so simple
- 11:47The Stream/Table Duality Table
- 12:24What is an Event Streaming Platform?
- 12:50Immutable Event Log
- 13:00Topics
- 13:44Partitions
- 14:23Messages are just K/V bytes
- 16:09Consumers have a position all of their own
- 18:10The Connect API
- 18:38Streaming Integration with Kafka Connect
- 18:51Stream Processing in Kafka
- 19:06Stream Processing with KSQL
- 20:17Events in Action
- 26:30Not Everything is a Nail
- 27:58Side-by-Side Tech Evaluation
- 28:54Evolve Data Sources
- 29:33Tight Coupling != Flexible
- 30:46Loose Coupling == Freedom to Evolve
- 31:55Transform Once, Use Many: Data Cleansing
- 32:45Transform Once, Use Many: Data Enrichment
- 33:38Message Payload Compatibility
- 34:41Build Resilient Pipelines with Schemas
- 40:17@rmoff
related talks
34mTales of Data Architecture Evolution
49mTransforming Analytical Data with Event-Driven Architecture
16mLife is But a (Data) Stream: Building Quality Data Pipelines
06mJay Kreps interview
43mA Data Streaming Architecture with Apache Flink #bbuzz
26mThe Tales of Event-Driven Told by a Spring Developer
01h 00mPutting Apache Kafka to Use
37mEvolution of Data Collection in the Datalake at Samokat Infrastructure