Ever Changing Data Model Schema Management For The Future
summary
Schema management centers on per-topic contracts: producers register schemas with a Confluent Schema Registry, send schema IDs with messages, and consumers use those IDs to deserialize, enabling replay and independent consumers on Kafka, AWS Kinesis or Google Pub/Sub. Demonstrated tools include Avro for schemas, Pact for consumer-driven contract tests, and stream processors like Kafka Streams, ksql, Flink and Beam. A concrete example shows adding a new parkingId field without a default causes the registry to reject it with a 409 incompatibility error, forcing explicit evolution rules.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:17The need for schemas
- 4:21Schemas to the rescue (Examples)
- 6:29Era of Web Services
- 9:39Contract Testing - Example
- 10:22Event Driven Architecture
- 12:51Event Streaming - A brief
- 17:29Schema Registry - How it works (Example)
- 19:18Schema Registry - How it works (Producer Example)
- 21:49Schema Registry - Summary
related talks
25mFrom Guesswork to Governance: Bringing API Discipline to Data Streaming
01h 00mEvolving Your Kafka Architecture With Schema Registry
38mStreaming with Structure
43mFrom Tower of Babel to Babel Fish: Evolving Kafka Architecture With Schema Registry
40mA Data Mesh Of Streams
27mLive in the Data Wild West: The Data Contracts Sheriff
26mKafka in the Wild
29mHigh-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale