Reliable Data Replication from Postgres on Kubernetes
summary
Datadog builds a scalable pipeline to replicate Postgres on Kubernetes to arbitrary targets using Debezium + Kafka Connect (preferring it over native logical replication for flexibility), with automation for provisioning, Datadog tracing for end-to-end visibility, and a schema registry to enforce compatibility. The platform reduces search load latency by 41% P90 on average and yields up to a 95% latency drop for some customers. A notable failure mode is Debezium's WAL bottleneck under heavy update spikes, which forced rollbacks.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 3:29Postgres for data replication
- 5:55Replication architecture
- 9:09Reliable replication requirements
- 10:35Logical replication for scalability
- 12:23Sync vs async replication
- 14:50Replication pipeline challenges
- 15:12Replication automation
- 17:43Metrics and traces
- 22:24Schema changes and DDL
- 23:48Schema registry checks
- 24:52Breaking change validation
- 26:20Customization and scalability
- 27:49Use case: 95% latency reduction
- 30:01Failed replication lessons
- 31:26Key takeaways
- 32:08Q&A: Migrations and downtime
- 34:00Q&A: Data transformations
- 35:03Q&A: Incremental snapshots
- 36:11Q&A: Pipeline reliability
- 37:35Q&A: Debezium guarantees
related talks
51mGeo-Distributed Postgres Replication Using Debezium
42mWhy PostgreSQL people should care about Kafka and Debezium?
26mBeyond the DBaaS Trap: Achieving Data Sovereignty With Kubernetes and CloudnativePG
33mTransitioning from Debezium ElasticSearch to Postgres for Streamlined Resilience
27mBuilding Event-Driven Systems with PostgreSQL Logical Replication and Drasi
25mPostgres on Kubernetes for the Reluctant DBA
41mStreaming your database - Easier said than done?
19mChange data capture with Debezium…and without