Scaling PostgreSQL Change Data Capture Using Open Source Airbyte
summary
Airbyte implements a WAL Acquisition Synchronization System that alternates partial snapshots with continuous WAL reads so long-running initial selects never exceed retention and no updates are lost. The system defaults to an eight-hour snapshot window, drains WALs between chunks, and finally converges to regular CDC; it handles multi-terabyte tables (Airbyte moved two terabytes on its hosted cloud) and uses Debezium plus reading from standbys to avoid primary load, realizing iterative snapshot + continuous WAL reads successfully.
AI-written summary · about
chapters (10)
- 0:00Airbyte and presenter intro
- 1:28What is Airbyte
- 2:11Change Data Capture basics
- 3:57Classic CDC replication
- 6:06Large-scale CDC challenges
- 7:51WAS algorithm solution
- 9:19PostgreSQL’s CDC role
- 11:02Airbyte’s connector kit
- 12:45Data integrity in CDC
- 13:55Row versioning and deduplication
related talks
35mChange Data Capture: The Magic Wand We Forgot #bbuzz
20mChange Data Capture in Distributed Systems
22mChange Data Capture: Exploring YugabyteDB’s Modern Design
19mChange data capture with Debezium…and without
01h 09mOpen-source Change Data Capture With Debezium
39mDeep Dive on Using YugabyteDB's Change Data Capture Connector
18mIncremental Backup in PostgreSQL
31mPostgreSQL WAL and Commit Synchronization, Optimization Opportunities