Developing Real-Time Data Pipelines with Apache Kafka
summary
Apache Kafka is described as a distributed commit log that can let a single cluster act as a central data backbone: a broker handles hundreds of MB/s from thousands of clients, typical configs use replication factor 3 with an ISR quorum and producer acks to guarantee durability, projects report 2 ms replication latency and benchmarked 2 million writes/sec on commodity machines. Ops examples include a Mesos-based scheduler for automated broker/partition management, a MirrorMaker pattern for cross‑DC replication, and a 25 ms‑class distributed RPC pattern that writes analyzed results back into Kafka for downstream consumers.
AI-written summary · about
chapters (32)
- 0:00Intro
- 0:28whoami
- 0:53Contents
- 1:56Apache Kafka
- 2:23How Big Data Usually Starts
- 2:52More Big Data!
- 3:58Kafka de-couples data pipelines
- 4:40Distributed Replicated Log
- 6:39Topics and Partitions
- 7:30Log Segments
- 10:28Data Durability
- 15:17Replication
- 23:33Producers
- 25:01Consumers
- 27:19Consumer Failover
- 31:15Producer Performance
- 33:01Consumer Performance
- 35:52Client Libraries
- 38:04Spring Integration
- 38:18Quick Start
- 39:02Operationalizing Kafka
- 41:03Running on Mesos
- 41:31Static Partitioning
- 42:15Scaling is manual (even if orchestrated)
- 42:30Static failures require manual intervention
- 42:41Application Elasticity
- 43:14An operating system for your data center
- 43:57Everything goes on Mesos
- 44:07Kafka on Mesos
- 49:21REST API & CLI
- 50:04Launch 20 brokers in seconds
- 50:17Kafka 0.9
related talks
50mIntroduction to Apache Kafka
38mKafka Message Broker Under High Load
53mDemystifying Kafka
41mApache Kafka simply explained
05mIntroduction to Kafka Internals
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design
33mKafka Microservices with Kafka plus Akka and Kafka Streams
48mLarge Scale, Distributed and Reliable Messaging with Kafka