CDC: From Zero to Hero
summary
Compares logical replication vs CDC and shows a concrete pipeline built inside double Cloud transfer that reads DB write‑ahead logs, streams events to Kafka, and feeds caches, full‑text indexes and replicas. First implementation serializes an internal event structure to JSON and breaks downstream consumers because JSON loses type metadata and binary vs text distinctions. Fixes include inventing a concise type system with type-system + versioning, adding compatibility tests, and adopting Debezium protocol extensions so the CDC pipeline finally works reliably.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:09Database basics and logs
- 3:18What is CDC and benefits
- 5:03CDC at Double Cloud Transfer
- 7:10Logical replication vs CDC
- 8:59First CDC serialization challenges
- 13:57Lessons from first CDC
- 16:23Second CDC with custom types
- 18:30Testing CDC implementations
- 19:31CDC serialization pitfalls
- 22:21Summary
related talks
38mCDC Without Pain: real-world resilient replication with Debezium and Kafka
24mDatabase Native Support for CDC Streaming Pipeline
41mStreaming your database - Easier said than done?
43mBack to fundamentals: from databases to event-driven platforms
35mImplementing CDC with change feed
32mA Change-Data-Capture use-case: designing an evergreen cache
30mChange Data Streaming Patterns in Distributed Systems
14mHow not(!) to build real-time apps