(Spring) Kafka - one more arsenal in a distributed toolbox
summary
Shows a real migration to Kafka with Spring Kafka and embedded Kafka for testing, using KafkaTemplate, @KafkaListener, KafkaMessageListenerContainer, KafkaTransactionManager and Spring-Kafka-test. Describes an event store of ~30 billion messages, 3 million reads per second and >500k writes/sec, explains zero-copy gains on Linux, log-structured storage, partition rebalancing, headers/typeId message conversion, how a chained JPA+Kafka transaction behaves (and why it's brittle), and demos that the setup works at scale.
AI-written summary · about
chapters (31)
- 0:00Introduction
- 0:42About me
- 1:20Lego
- 2:17Kafka overview
- 3:59Kafka guarantees
- 7:31What is Kafka
- 8:50Challenges
- 10:00Opportunity
- 12:32Kafka embrace immutability
- 14:23Kafka log structure
- 14:52Zero copy operation
- 16:58Multiple machines
- 17:42Fast writes
- 18:36Resilience
- 19:51Spring Kafka
- 21:40Demo
- 29:10Demo breakdown
- 29:30Kafka consumer factory
- 35:44Kafka listener
- 36:44Kafka consumer
- 40:44KafkaListener
- 42:44Message Converter Beam
- 43:44Type ID
- 44:44Cancel Exclusion
- 45:44Chain Transaction Manager
- 46:44Order Service
- 47:44Thread of Sleep
- 48:44Uncommitted Messages
- 49:44Santu
- 50:44EventListener
- 52:44Replay Time
related talks
41mApache Kafka simply explained
53mDemystifying Kafka
50mIntroduction to Apache Kafka
33mKafka Microservices with Kafka plus Akka and Kafka Streams
18m"Kafka's Metamorphosis"
38mKafka Message Broker Under High Load
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design