Distributed DDL Replication at Global Scale
summary
Nile implements virtual tenant Postgres databases across multiple AWS regions and replicates DDLs by hooking an extension into the process utility hook, sending parsed and transformed statements to a Rust transaction coordinator over protobuf, and using Postgres prepared transactions (2PC) to commit. Locks are extracted from statement behavior and taken deterministically (including a user‑tag advisory lock workaround for objects that don’t exist). The system retries/fences on commit ambiguity and currently supports Postgres 13–17 while excluding functions and some partitioning features.
AI-written summary · about
chapters (17)
- 0:00Introduction and speaker background
- 2:35Why distributed DDL was needed
- 6:13Virtual tenant databases explained
- 8:02How the solution works
- 13:57Handling transactions in distributed DDL
- 20:28Locking challenges and solutions
- 27:33Failure handling and eventual consistency
- 34:37Summary and future plans
- 36:04Q&A: Resolving prepared transactions
- 37:54Q&A: Online migrations in PostgreSQL
- 40:00Q&A: Retry behavior for failed nodes
- 41:23Q&A: Node recovery and consistency
- 42:28Q&A: Lost commit messages scenario
- 43:11Q&A: User access and virtual databases
- 44:16Q&A: Lock extraction and PostgreSQL updates
- 46:25Q&A: Unsupported DDL types
- 47:48Q&A: Transaction coordinator HA
related talks
46mpg paxos Table Replication through Distributed Consensus
38mPostgreSQL as a distributed computing platform
53mMulti master replication for painless fault tolerance
24mAutomatic DDL Replication with pgEdge Distributed Postgres
49mPostgreSQL’s Journey to Distributed Databases
37mDistributed SQL Transactions
42mPostgreSQL Distributed: Architectures & Best practices
24mNavigating Distributed Databases: Challenges & Opportunities