Designing Payloads for Event-Driven Systems
summary
Design payloads for Kafka by sending non-essential metadata in headers, using the key to shard/order across partitions, and keeping the root an object — never top-level array. Keep messages small (default broker limit ~1MB) and include a payload-level timestamp in ISO with timezone — include ISO timestamp. Use Avro or protobuf with a schema registry (Carapace or Confluent) for compatibility and compression, emit OpenTelemetry trace/span ids, and publish AsyncAPI + Avro refs to generate docs and Python/Go producer/consumer code.
AI-written summary · about
chapters (10)
- 0:00Introduction to event-driven systems
- 3:21Kafka payload design basics
- 7:21Structuring payload data
- 12:28Timestamps and event tracing
- 16:22Choosing data formats and schemas
- 20:17Schema evolution and compatibility
- 22:04Documenting payloads for humans
- 22:48AsyncAPI for event-driven systems
- 25:59Conclusion and key takeaways
- 28:27Q&A on message formats
related talks
28mDesigning Payloads for Event-Driven Systems
46mMake Your Kafka Applications More Robust
38mStreaming with Structure
29mGo Big with Apache Kafka
41mKafka as a Platform: The Ecosystem from the Ground Up
43mBuilding Scalable and Extendable Data Pipeline
45mReliable Toolkit for Event-Driven Microservice Architecture
55mAvoiding mistakes with events, one event at a time