Spring Kafka beyond the basics - Lessons learned on our Kafka journey at ING Bank

on YouTube ↗44m

" summary

ING runs Kafka in production for 7+ years with ~5,000 topics, ~500k msgs/sec and >800 dev teams; demos show a local cluster + schema registry, console producer creating poison pills, and spring-kafka fixes that use the ErrorHandlingDeserializer and publishing to a dead letter topic to avoid log storms. Exception handling uses configurable retries and backoffs. Testing shifts from EmbeddedKafka to the Kafka Streams topology test driver and Testcontainers. Metrics use Micrometer + Prometheus and distributed tracing via Sleuth/Zipkin.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 3:47Kafka in a nutshell
3. 7:44Spring for Apache Kafka
4. 13:35Scenario: Poison Pill (Deserialization exception)
5. 25:43Scenario: Lack of proper exception handling
6. 32:18Testing
7. 37:30Monitoring
8. 42:12Lessons learned

" related talks

• 50mSpring Kafka beyond the basics - Kafka journey at ING BankTim van Baarsen · J-Fall 2022
• 46mSpring Kafka. How to Prepare and Not Shoot Yourself in the FootIvan Golovko · Joker 2024
• 43mEvent streaming applications with Kafka Streams, Spring Kafka and ActuatorKo Turk · JNation 2022
• 52mStream Processing with the Spring Framework Like You’ve Never Seen It BeforeJosh Long, Viktor Gamov · Spring I/O 2019
• 51mStateful & Reactive Stream Processing Applications Without a DatabaseHans-Peter Grahsl · Voxxed Days Ticino 2018
• 51mEvent Driven with SpringOleg Zhurakousky · Spring I/O 2019
• 01h 02mMore Practical Tips and Tricks with Spring IntegrationOleg Zhurakousky · SpringOne2GX 2012
• 01h 08mSpring Integration Done Boot-ifullyGlenn Renfro · SpringOne2GX 2014