From Guesswork to Governance: Bringing API Discipline to Data Streaming

on YouTube ↗25m

" summary

Claim: move validation upstream so producers can’t write poison data and consumers can trust event formats. Uses schema registry (stored in Kafka), schema-aware serializers that check compatibility and block incompatible writes, and popular formats like Apache Avro (also supports Google Protobuf). Demo registers a schema in Confluent Cloud, produces 10 events, bumps a major schema version, registers upgrade/downgrade transforms, and consumers successfully read v1 events as v2, proving evolution and enforcement work.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 2:30APIs need standards
3. 5:57Kafka’s byte storage
4. 8:51Fire-and-forget without contracts
5. 10:57Schema evolution and governance
6. 13:23Schema registry as truth
7. 17:54Schema-aware serializers
8. 19:59Live demo: schema evolution
9. 25:02Q&A

" related talks

• 38mStreaming with StructureKatherine Stanley · GOTOpia February 2021
• 23mEver Changing Data Model Schema Management For The FutureVidya Venugopal · DataEngBytes 2020
• 31mOne Data Pipeline to Rule Them AllSam Kitajima-Kimbrel · PyCon AU 2017
• 01h 00mEvolving Your Kafka Architecture With Schema RegistryViktor Gamov · IntelliJ IDEA Conf 2025
• 43mFrom Tower of Babel to Babel Fish: Evolving Kafka Architecture With Schema RegistryViktor Gamov · Voxxed Days Thessaloniki 2025
• 39mMoving Beyond Moving BytesSuneel Marthi, Joey Frazee · Flink Forward Berlin 2017
• 29mHigh-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix ScaleJagannathrao Mudda, Ramayan Tiwari · Flink Forward Virtual 2020
• 23mThe Enterprise Guide To Building A Data MeshSion Smith, Neil Avery · Big Data LDN 2022