Navigating Streaming Infrastructure
summary
Compares concrete tradeoffs: Kafka as immutable, partitioned log with replicate three times for durability; producers serialize, consumers control exactly‑/at‑least‑/at‑most‑once guarantees; Kafka Connect cuts custom connector work from six months to two weeks in practice; Flink provides built‑in stateful stream processing and dynamic tables with tumbling/sliding/session windows, while Spark maps to batch/RDD semantics; recommends schema registry plus dead‑letter topics for invalid messages and shows these patterns running end‑to‑end.
AI-written summary · about
chapters (15)
- 0:00Welcome
- 1:49Data systems evolution
- 4:39Data streaming and Kafka
- 6:23Companies as software
- 9:02Data engineering challenges
- 10:07Streaming infrastructure pillars
- 11:52Kafka architecture basics
- 14:18Data durability with replication
- 16:00Producer internals and serialization
- 17:03Real-time processing with Flink and Spark
- 21:59Flink cluster and dynamic tables
- 23:23Windowing strategies
- 24:27Kafka Connect
- 25:09Streaming governance and schemas
- 26:36Summary
related talks
45mApplications in the Emerging World of Stream Processing
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
44mData Streaming: Which horse to bet on?
47mOne Does Not Simply Query a Stream
01h 00mIn Depth Kafka Streams
55mDemystifying Stream Processing with Apache Kafka
48mOne Does Not Simply Query a Stream
39mEvaluating Streaming Framework Performance for a Large-Scale...