Putting Apache Kafka to Use
summary
Experience moving LinkedIn from batch ETL to a real‑time event platform using Apache Kafka: a commit log design (partitioned, replicated topics) that preserves strict ordering and durable retention SLAs, feeding Hadoop/warehouses and real‑time processors like Storm Spark Samza. Operations run per data center with handfuls of clusters, storing about a petabyte of stream data and handling roughly 800 billion messages written per day; the system proves practical and scalable for both ETL and low‑latency stream processing.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 5:36Problems with data pipelines
- 13:38Why existing messaging systems failed
- 16:28Introducing Apache Kafka
- 26:53Stream processing and real-time data
- 38:06Kafka at LinkedIn scale
- 41:26Handling cluster failures
- 53:15Q&A session
related talks
50mIntroduction to Apache Kafka
05mPREVIEW: 1 Day, 1 Data Hub, 100 Billion Messages
50mApplication development and data in the emerging world ...
26mLeveraging Data in Motion in a Cloud-first World
38mKafka Message Broker Under High Load
41mKafka as a Platform: the Ecosystem from the Ground Up
16mAn introduction to Apache Kafka - this is Event Streaming, not just Messaging
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...