Evolving Your Kafka Architecture With Schema Registry
summary
Demonstrates turning Schema Registry into the polyglot backbone for Kafka by running a Docker Compose demo with a Kotlin/Java order service, Python inventory worker and NodeJS analytics, using Avro schemas, generated language bindings, and a schema subject per topic. Schema Registry stores schemas in Kafka topic, enforces backward compatibility (adds fields succeed, breaking removals are rejected), and integrates into CI/CD via code generation and testcontainers; the demo works end to end.
AI-written summary · about
chapters (12)
- 0:00Introduction to Kafka challenges
- 3:08Data format challenges
- 6:06Demo architecture overview
- 9:31Java and JSON serialization issues
- 13:22The need for data contracts
- 18:41Schema registry as a solution
- 23:58Avro and schema implementation
- 36:04Communication and schema versioning
- 43:34Schema compatibility management
- 50:16CI/CD and schema governance
- 53:51Testing with Testcontainers
- 57:17Handling breaking changes
related talks
43mFrom Tower of Babel to Babel Fish: Evolving Kafka Architecture With Schema Registry
41mKafka as a Platform: The Ecosystem from the Ground Up
23mEver Changing Data Model Schema Management For The Future
39mKafka as a Platform: The Ecosystem from the Ground Up
38mStreaming with Structure
15mAn Empty Kafka is a Sad Kafka
25mFrom Guesswork to Governance: Bringing API Discipline to Data Streaming
41mKafka as a Platform: the Ecosystem from the Ground Up