Apache Kafka and the Next 700 Stream Processing Systems
summary
Apache Kafka provides a partitioned commit log, log compaction, consumer groups and replayable offsets to make stateful, fault tolerant stream processing practical at datacenter scale. Frameworks integrated include Samza, Spark Streaming, Storm and Flink and a lighter-weight library is Kafka Streams that journals local state to Kafka (often backed by RocksDB). The system runs as LinkedIn’s central event hub and handles about 1.1 trillion messages per day, enabling reprocessing, scalable parallelism and exactly the persistence guarantees needed for joins and late‑arriving data.
AI-written summary · about
chapters (11)
- 0:00Start
- 7:33PROCESSING THE
- 8:43RETAIL
- 11:29STREAM PROCESSING
- 16:02LOGS & STREAMS
- 18:04CHANGE LOGS
- 19:19COMPACTION
- 20:49KAFKA AND THE STREAM
- 21:57PARTITIONING
- 25:50REPROCESSING
- 26:29STREAM DATA PLATFORM
related talks
31mEvent Streaming with Kafka Streams
34mChasing the Stream Processing Utopia
33mKafka streams at scale
17mOvercome By Events - Apache Kafka And Event-oriented Architecture
53mDemystifying Kafka
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
42mDistributed Real Time Stream Processing: Why and How
13mKeynote – Jay Kreps