A journey into postgresql logical replication

on YouTube ↗38m

" summary

Built an in-house CDC pipeline using Postgres logical replication and a Golang logical replication client, the team finds that two wrong assumptions cause production data loss: LSNs are not globally incremental under concurrency, and acknowledging only data-changing events (not transaction end) lets restarts replay or skip needed changes. A custom WAL parser and discussions with Postgres devs lead to the rule commit only on transaction end; that fixes loss but forces transaction batching and splitting to avoid messaging-size and duplication issues.

AI-written summary · about

" chapters (9)

1. 0:00Motivation for custom CDC
2. 2:22Building the Golang client
3. 5:13Understanding LSN offsets
4. 8:56Concurrency and replication
5. 13:09Common replication pitfalls
6. 18:16Correcting replication errors
7. 22:28Refined replication approach
8. 26:13Handling capacity constraints
9. 29:59Q&A and concluding remarks

" related talks

• 29mLogical replication theory and conceptsAshutosh Bapat · POSETTE: An Event for Postgres 2025
• 42mlesser known Superpower of Postgres Logical ReplicationKevin Biju · PGConf India 2025
• 53mThe journey towards active replication in PostgreSQLJonathan S. Katz, Amit Kapila · PGCon 2023
• 59mLogical Replication...LIVE!Robert Treat · PostgresOpen Silicon Valley 2017
• 56mLogical Replication of DDLsPeter Smith, Zheng Li (Zane) · PGCon 2023
• 31mPast, Present, and Future: Logical Decoding and Replication in PostgreSQLHari P Kiran · POSETTE: An Event for Postgres 2026
• 54mUnderstanding Logical Decoding and ReplicationMichael Paquier · PostgresOpen 2014
• 22mSecuring Postgres with streaming replicationJan Karremans · POSETTE: An Event for Postgres 2025