Re-architecting Apache Kafka for the Cloud
summary
Apache Kafka adds tiered storage that uploads ~100 MB segments to S3/HDFS so clusters keep only hot segments locally, cutting data movement and making broker add/remove operations take minutes instead of days; a new KRaft controller replaces ZooKeeper, stores metadata in Kafka logs, runs as a single binary and reduces full-state reloads; built-in multi-tenant quotas and namespace plugins isolate tenants and apply backpressure; claim is cluster scale moves from hundreds of thousands toward millions of partitions; KRaft is available in 2.8 but not yet production-ready.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:15Cloud native data systems
- 1:56Cloud native databases
- 3:04What is cloud native
- 4:29Conceptual integrity
- 5:41Log obstruction
- 7:23Companywide scale
- 9:08Elasticity
- 10:19Three key changes
- 10:58Tiered storage
- 11:36Tiered storage explained
- 13:21Tiered storage benefits
- 14:26Cache separation
- 15:19New controller
- 18:08Kafka replicas
- 21:06Kafka multitenant
- 22:08What is multitenant
- 22:50Confluent Cloud
- 23:36Kafka Plugins
- 27:47New Kafka
- 28:51Questions
- 31:18Kip
- 33:30Kafka on AWS
- 35:01Kafka on Craft
related talks
02h 04mKafka with Raft in Production: A New Era for Distributed Commit Logs
26mReleased From the Cage: Apache Kafka Without Its ZooKeeper
50mIntroduction to Apache Kafka
24mMake Your Kafka Cluster Production-Ready
53mDemystifying Kafka
29mApache Kafka and the Next 700 Stream Processing Systems
50mKafka 4, fantastic?
44mKafka 4, Fantastic?