Demystifying Kafka
summary
Explains how Apache Kafka achieves high throughput and resiliency with concrete mechanisms: partition-based scaling and automatic partition rebalancing; log-structured storage using OS page cache and zero-copy kernel IO to avoid JVM buffering; replication with configurable replication factor 3 and ISR semantics; idempotent producers and a transaction coordinator using marker-based transactions plus consumer read_committed isolation to avoid ghosts; practical numbers (Netflix ~7 trillion messages/day), JVM heap guidance (4–6 GB), Kafka Connect, Streams and ksql, and common production pitfalls like shared disk contention.
AI-written summary · about
chapters (16)
- 0:00Introduction and speaker intro
- 1:16Why Kafka over traditional messaging
- 4:06Kafka as more than a messaging system
- 10:06Kafka vs message-oriented middleware
- 14:05Immutability and log structure in Kafka
- 17:05Kafka scalability and throughput
- 22:01Kafka caching and performance
- 23:45Exactly once processing in Kafka
- 27:39Kafka transactions explained
- 33:06Kafka replication and resiliency
- 40:47Data streaming with Kafka Connect
- 44:19Kafka Streams and KSQL
- 46:47Schema registry and data compatibility
- 49:17Kafka as a database inside out
- 50:19Kafka deployment best practices
- 52:58Summary and Q&A
related talks
41mApache Kafka simply explained
05mIntroduction to Kafka Internals
39mKafka as a Platform: The Ecosystem from the Ground Up
50mIntroduction to Apache Kafka
41mKafka as a Platform: The Ecosystem from the Ground Up
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
38mKafka Message Broker Under High Load