Change Data Capture Pipelines with Debezium and Kafka Streams
summary
Debezium tails database transaction logs and publishes low‑latency change events to Kafka (often sub‑second); connectors exist for MySQL, Postgres, MongoDB and SQL Server. Kafka Streams runs in a Quarkus runtime to build stateful pipelines—demo shows hot‑reload Quarkus dev mode enriching IoT measurements with master data, aggregating averages, and storing state in RocksDB. KIP‑213 enables non‑key joins to materialize nested aggregates; transaction boundary markers and buffering let pipelines emit complete aggregates only after all related events arrive.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 2:08What is Debezium
- 8:12Debezium use cases
- 12:01Kafka Streams and Quarkus
- 17:20Data enrichment demo
- 24:09Auditing with change events
- 29:02Handling partial updates
- 34:06Aggregate view materialization
- 44:18Summary and resources
related talks
42mData Streaming for Microservices using Debezium
41mChange Data Streaming Patterns in Distributed Systems
01h 09mOpen-source Change Data Capture With Debezium
31mChange Data Capture for non relational databases
48mStreaming Database Changes with Debezium
23mChange Data Capture: A RabbitMQ and Debezium Approach
52mChange Data Streaming For Spring Boot With Apache Kafka and Debezium
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!