Spring Kafka beyond the basics

on YouTube ↗50m

" summary

ING runs Kafka for 7+ years with over 5,000 topics, more than 1,000 developer teams, and ~1M messages per second growth ~10% monthly. Demonstrations use Spring Kafka + Confluent Schema Registry, Kafka Streams, Prometheus/micrometer and Zipkin, plus local tooling (conductor, embedded Kafka, Testcontainers). Shows concrete fixes: configure ErrorHandlingDeserializer to avoid infinite retry poison pills and publish bad records to a dead-letter topic, use the topology test driver for fast Kafka Streams unit tests, and add metrics/tracing to spot consumer lag.

AI-written summary · about

" chapters (13)

1. 0:00Introduction and Kafka at ING
2. 12:32Local Kafka setup and tools
3. 13:14Poison pill scenario explained
4. 16:47Fixing poison pill with Spring Kafka
5. 21:44Dead letter topic recovery
6. 26:41Exception handling in consumer code
7. 29:29Demo of consumer exception handling
8. 31:40Testing Kafka applications
9. 33:54Kafka Streams unit testing
10. 37:24Monitoring Kafka with metrics
11. 38:50Distributed tracing demo
12. 43:30Lessons learned and wrap-up
13. 45:41Q&A session

" related talks

• 48mSpring for Apache Kafka the advanced featuresTim van Baarsen, Kosta Chuturkov · Spring I/O 2025
• 01h 29mEvent stream processing using Kafka StreamsFredrik Vraalsen · Berlin Buzzwords 2018
• 40mEvent Streaming Applications with Kafka Streams, Spring Kafka and ActuatorKo Turk · Voxxed Days Bucharest 2022
• 01h 44mData processing with Kafka Streams and Spring Framework.Dmitry Sergeev · Devoxx Ukraine 2020
• 01h 11mBootiful Stream Processing!Viktor Gamov · Java Z-Day 2021
• 50mStream processing with Kafka in Big Data environmentsEvgeniy Nenakhov · Joker 2023
• 57mWhen Everything Went KafkaGrigoriy Koshelev · JPoint 2019
• 27mStream processing with Kafka streamsHamza Belmellouki · BlaBlaConf 2022