Change Data Capture in Distributed Systems
summary
Compares three practical CDC patterns — polling with status timestamps (simple but wasteful and brittle), triggers/store procedures (low-latency but DB-resource heavy), and reading the transaction log (no DB impact, fast, but vendor-specific). Describes a working pipeline that reads Postgres WAL and uses AWS DMS to stream into DynamoDB for read-side caches, with services implemented in Play/Doobie and clients querying cached tables; notes this decouples services, scales better than polling, and cites open-source tools like Debezium and Maxwell as alternatives.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:37Source of Data
- 2:37CDC
- 3:14Reliability
- 4:22Strategies
- 6:55Implementing CDC
- 12:27Use Cases
- 15:55Frameworks
- 19:44Outro
related talks
35mChange Data Capture: The Magic Wand We Forgot #bbuzz
22mChange Data Capture: Exploring YugabyteDB’s Modern Design
47mCDC Streaming Patterns in Distributed Systems
15mScaling PostgreSQL Change Data Capture Using Open Source Airbyte
45mModern Data Architecture
19mChange data capture with Debezium…and without
01mA Change-Data-Capture use-case: designing an evergreen cache
48mApplication Architectures: Architecture concepts based on CDC