Change Data Capture: The Magic Wand We Forgot #bbuzz
summary
Change Data Capture extracts every committed row-level change from PostgreSQL 9.4 via logical decoding and streams it into Kafka using a small in-server plugin plus a client daemon that takes a consistent MVCC snapshot then catches up on live commits. Tables map to Kafka topics, message key = primary key, updates carry the full new row, deletes become a null value compatible with log compaction, and schemas are emitted as Avro schemas. Demo shows end-to-end latency around one second; code is open source alpha.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 5:42What is change data capture
- 8:51Introducing Bottled Water
- 12:44Live demo of Bottled Water
- 15:13Consistent snapshots and streaming
- 18:02Handling transactions and concurrency
- 20:50How Kafka works with CDC
- 22:34Postgres to Kafka mapping
- 29:04Postgres internals and logical decoding
- 33:39Other CDC implementations and resources
- 34:43Q&A
related talks
01h 09mOpen-source Change Data Capture With Debezium
22mChange Data Capture: Exploring YugabyteDB’s Modern Design
15mScaling PostgreSQL Change Data Capture Using Open Source Airbyte
19mChange data capture with Debezium…and without
23mChange Data Capture: A RabbitMQ and Debezium Approach
31mChange Data Capture for non relational databases
39mDeep Dive on Using YugabyteDB's Change Data Capture Connector
20mChange Data Capture in Distributed Systems