10 years of Kafka. Tips, tricks and lessons learned
summary
Kafka runs globally for 10 million connected cars with thousands of topics and billions of messages per day, and concrete solutions drive the value: a copycat (500 lines) replicates topics across clusters and enables a seamless 500‑app migration, an OAuth Kafka proxy (Croxylicious) lets Volvo ID use mTLS without changing clusters, an MQTT→Kafka bridge (≈50 pods, ~$100/day, <1/10 cost vs IoT Core) handles millions of persistent car connections, a gRPC consumer bridge reduces partitions and rebalances, and Protobuff replaces schema registry for simpler compatibility.
AI-written summary · about
chapters (12)
- 0:00Kafka at Volvo Cars
- 5:04Apache Kafka basics
- 8:55Kafka scalability
- 11:02Global replication components
- 14:35OAuth Kafka proxy
- 17:08gRPC consumer bridge
- 20:37MQTT Kafka bridge
- 24:08Kafka Streams views
- 28:27Self-hosted vs managed Kafka
- 31:38Migration to managed Kafka
- 34:51Regulations for Kafka topics
- 38:21Q&A
related talks
52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICES
30m8 Lessons Learned from using Kafka with 1000 microservices
01h 00mKafka for .NET Developers
31mStreaming: Real-Time Architecture for Connected Products in Automotive
21mHow Apache Kafka® Generates Value In Retail With Confluent
54mMicroservices with Kafka
51mFive years of highs and lows with Kafka
54mKafka: The Must Know Pub/Sub