Streaming your database - Easier said than done?
summary
A contractor implements a Change Data Capture (CDC) pipeline over 18 months that connects Postgres replication to Kafka topics, layers an enrichment step, and feeds cloud RDS and caches. The talk gives exact failure modes: 40k-row transactions that crash consumers, choice of replication format, massive duplicate messages (>90% hits on an LRU cache), partitioning keys to preserve order, bulk enrichments of ~500 messages, and dashboards/metrics that prove which fixes actually reduce latency, cost, and data loss.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:09Change data capture basics
- 3:59Legacy CDC problems
- 5:50Why Debezium wasn’t the solution
- 9:07New CDC architecture design
- 12:21Unexpected benefits
- 13:24Legacy problems solved
- 15:13Implementation steps summary
- 16:37Kafka for multi-tenant clusters
- 18:45Why nothing worked in production
- 19:27Pitfalls to avoid
- 23:25Postgres resilience handling
- 26:38Monitoring Kafka offsets
- 27:01Data filtering for performance
- 28:46Slowing RDS writes
- 29:53Duplicate data detection
- 31:18Kafka enrichment optimization
- 32:20Kafka partitioning best practices
- 34:28Metrics and monitoring importance
- 36:59Key lessons learned
- 38:43Baseline data handling
- 40:33Conclusion and Q&A
related talks
30mChange Data Streaming Patterns in Distributed Systems
35mChange Data Capture: The Magic Wand We Forgot #bbuzz
38mCDC Without Pain: real-world resilient replication with Debezium and Kafka
27m"Write Once Stream Everywhere: Transforming Your Data into Events"
23mChange Data Capture: A RabbitMQ and Debezium Approach
31mZero to data infrastructure
39mRethinking RDBMS Data Migration
43mBack to fundamentals: from databases to event-driven platforms