When Everything Went Kafka

on YouTube ↗57m

" summary

A practical look at running Apache Kafka: the talk examines Kafka component architecture and collects common operational pitfalls (misleading defaults, client-side surprises, missing docs). It explains the architectural trade-offs behind those issues, discusses why DevOps involvement is necessary, and evaluates whether Kafka’s many configuration options are helpful or harmful.

AI-written summary · about

" chapters (26)

1. 0:00Introduction
2. 1:05Why Kafka in projects
3. 3:12Kafka components
4. 7:27Cluster architecture and replication
5. 10:14Producer internals
6. 14:26Consumer internals
7. 17:12Controller role
8. 18:39Kafka pitfalls
9. 20:46Split-brain data loss
10. 21:07Replication and leadership issues
11. 22:53Default replication settings
12. 24:19Broker-client config mismatch
13. 25:21Message size errors
14. 26:45Batch size vs latency
15. 28:31Blocking send issues
16. 29:35Consumer driver errors
17. 31:41Multi-partition reading
18. 34:11Manual disk balancing
19. 36:39Topic size limits
20. 37:24Zero-copy and compression
21. 40:45Finding data in Kafka
22. 42:53Timestamp problems
23. 44:42Retention and segment errors
24. 46:28Consumer offset issues
25. 50:20Key takeaways
26. 52:13Q&A

" related talks

• 54mWhen Everything Went Kafka: Apache Kafka and the ConsumerGrigoriy Koshelev · JPoint 2023
• 16mIntroducing Apache KafkaKatherine Stanley · Devoxx Belgium 2019
• 46mEmbracing AnarchyRicardo Ferreira · QCon São Paulo 2019
• 57mDeep Dive into Kafka Architecture: From Simple Scenarios to a Geo-ClusterAndrey Komyagin · Saint HighLoad++ 2025
• 46mBrick-by-Brick: Exploring the Elements of Apache Kafka®Danica Fine · Berlin Buzzwords 2024
• 51mFive years of highs and lows with KafkaNelson Dionisi, Matthieu Mouminoux · Codeurs en Seine 2022
• 39m7 Years in Kafka LandFabien ALBERI, Valérie Servaire · Devoxx France 2026
• 27mKafka - How a Tortoise Helps Us Break ChainsFlorian Albrecht, Gunnar Kiesel · code.talks 2022