Open-source Change Data Capture With Debezium
summary
Debezium is an open source CDC platform that captures database transaction logs via bespoke connectors (Postgres, MySQL, MongoDB, Oracle) and emits change events into Kafka or other systems using Kafka Connect, an embedded engine, or Debezium Server. It claims end-to-end latency <2s for many databases, exposes schema history in a Kafka topic, handles the outbox pattern for reliable messaging, and implements incremental snapshotting with watermarks to backfill large tables; main pain points are DDL parsing and push vs poll API mismatches.
AI-written summary · about
chapters (22)
- 0:00Introduction to Debezium
- 1:05Change data capture basics
- 3:38Use cases for CDC
- 6:17How Debezium works
- 8:42Advanced CDC capabilities
- 10:55Connector design and formats
- 13:13Connector framework
- 15:13Internal architecture
- 18:14Running Debezium
- 21:00Handling transactions
- 23:08Event structure
- 25:45The outbox pattern
- 30:49Outbox implementation
- 33:52Postgres logical messages
- 38:21Schema management challenges
- 43:23Efficient schema handling
- 46:33Oracle log mining
- 48:55Snapshotting techniques
- 51:52Incremental snapshots
- 1:02:38Future of CDC and conclusion
- 1:05:14Q&A session
- 1:08:53Final discussion
related talks
23mChange Data Capture: A RabbitMQ and Debezium Approach
31mChange Data Capture for non relational databases
48mStreaming Database Changes with Debezium
41mChange Data Streaming Patterns in Distributed Systems
49mChange Data Capture Pipelines with Debezium and Kafka Streams
38mCDC Without Pain: real-world resilient replication with Debezium and Kafka
26mDebezium Server - New CDC runtime on the horizon!
39mDeep Dive on Using YugabyteDB's Change Data Capture Connector