Beginners Guide to Balance Your Data Across Apache Kafka Partitions
summary
Explains how Kafka decides partitions: explicit partition, key-hash, or sticky partitioning (round‑robin replaced; benchmarks show ~50% latency reduction and 5–15% CPU drop), why pick partition count from consumer throughput needs, and why partitions cannot be reduced later without rework. Covers hot‑partition failure modes, using high‑cardinality keys (example: purchase flow ID instead of customer ID), when to add partitions or recreate topics, and the Kafka 3.3 options like adaptive partitioning and availability timeout.
AI-written summary · about
chapters (17)
- 0:00Introduction to Apache Kafka
- 1:25Understanding Kafka partitioning
- 3:58Key metrics in Kafka performance
- 6:47Choosing partition count for topics
- 10:40Balancing partitions and scalability
- 13:08Default partitioner behavior
- 16:21Sticky partitioning explained
- 19:11Monitoring partition distribution
- 20:58Risks of unbalanced partitions
- 27:05Selecting effective keys
- 31:58Handling slow brokers
- 35:37Scaling partitions and consumers
- 37:04Proactive monitoring strategies
- 38:27Fixing hot partitions
- 42:04Key learnings and advice
- 43:31Ivan platform and resources
- 57:19Q&A session
related talks
47mBeginners guide to balance your data across Apache Kafka partitions
46mBrick-by-Brick: Exploring the Elements of Apache Kafka®
36mApache Kafka Simply Explained
50mIntroduction to Apache Kafka
54mKafka: The Must Know Pub/Sub
52m15 Essential Do’s and Don’ts with Apache Kafka
46mMake Your Kafka Applications More Robust
16mAn introduction to Apache Kafka - this is Event Streaming, not just Messaging