Apache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design
summary
Claims LinkedIn scales Kafka to 4.4 million writes per second across ~350 brokers and handles 278 billion messages per day. Design specifics: Kafka acts as an append-only write‑ahead log with per‑partition offsets, uses sequential disk I/O and OS page cache, batches and compresses records, and serves consumers with zero‑copy sendfile transfers to reduce CPU. Zookeeper manages cluster coordination and consumer group offsets, allowing rewindable offsets. Demo wires a Twitter stream into Kafka, consumes with Spring Boot threads, exposes counts via a simple AngularJS UI.
AI-written summary · about
chapters (21)
- 0:00Intro
- 5:20Application Design
- 10:10What is Kafka
- 14:07Kafka Consumer
- 14:58Mechanical Sympathy
- 16:05Read Ahead
- 16:52Batch Push
- 18:39Zero Copy
- 19:48Kafka by Design
- 21:25Kafka vs Spark
- 24:05Message Rate
- 27:16Topic
- 28:08Partition
- 30:03Partition logic
- 31:35Batch
- 33:05Consumers
- 34:29Consumer Message Editor
- 38:09Message Delivery Guarantee
- 39:59Final Messages
- 40:55Regular Use Cases
- 41:59Extreme Processing
related talks
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
38mData Pipeline with Apache Kafka
53mDemystifying Kafka
01h 00mPutting Apache Kafka to Use
38mKafka Message Broker Under High Load
41mApache Kafka simply explained
53m(Spring) Kafka - one more arsenal in a distributed toolbox
50mIntroduction to Apache Kafka