Data Streaming for Microservices using Debezium
summary
Debezium streams database change logs into Apache Kafka so services get ordered, complete change events (before/after + source metadata) without code changes; connectors exist for MySQL, PostgreSQL and MongoDB and the demo wires Kafka Connect, a 3‑node Kafka cluster and Strimzi on OpenShift, runs a Kafka Streams aggregation over orders by category and pushes results over WebSockets — the live pipeline runs end‑to‑end. Key details: messages use primary key routing, topics can be compacted, and Kafka Connect offsets and SMTs handle resumption and lightweight transforms.
AI-written summary · about
chapters (16)
- 0:00services Using
- 1:26Change Data Capture What is it about?
- 2:12CDC Use Cases
- 6:06How to Capture Data Changes!
- 11:24Apache Kafka Perfect Fit for CDC
- 16:35CDC Topology with Kafka Connect
- 17:40CDC Message Structure
- 19:48Debezium Connectors
- 20:41Pattern: Microservice Data Synchronization Microservice Architectures
- 23:11Pattern: Microservice Extraction Migrating from Monoliths to Microservices
- 24:28Pattern: Materialize Aggregate Views
- 27:24Pattern: Ensuring Data Quality Detecting Missing or Wrong Data
- 29:19Pattern: Leverage the Powers of SMTS Single Message Transformations
- 38:22Debezium Current Status
- 39:51Summary
- 40:29Resources
related talks
15mManaging Data Consistency among Microservices with Debezium
48mStreaming Database Changes with Debezium
52mChange Data Streaming For Spring Boot With Apache Kafka and Debezium
41mChange Data Streaming Patterns in Distributed Systems
49mChange Data Capture Pipelines with Debezium and Kafka Streams
55mThe hardest part of microservices: Your Data
01h 09mOpen-source Change Data Capture With Debezium
31mKafka and Debezium at trivago