Move Less, Move Faster: Speeding Up Citus Cluster Scaling
summary
A real-world 18–20 TB Citus rebalance that used to take 3–4 days gets two concrete changes: a parallel rebalancer that copies reference tables concurrently, relaxes overly conservative locks so multiple shard moves run simultaneously, and exposes flags like parallel_transfer_reference_tables (default off) for tuning; and a snapshot-based node addition that boots a new worker from a base backup/disk snapshot, attaches it via citus_add_clone_node and completes by running citus_promote_clone_and_rebalance, turning expensive row-by-row moves into fast deletes (benchmarks show ~26–30% gains).
AI-written summary · about
chapters (13)
- 0:00Music & introduction
- 0:22“Move Less, Move faster” overview
- 0:46Real-world slow rebalance example
- 2:33Citus basics and architecture
- 4:15How rebalancing works in practice
- 5:44Why rebalancer performance was slow
- 8:21Parallelising reference table copies
- 10:45Fixing locking for parallel shard moves
- 14:17Orchestrating a parallel rebalancer
- 22:15Snapshot-based scaling: move less
- 24:59Snapshot-based node addition
- 27:30Faster rebalance by deleting instead of moving
- 29:31Choosing the right scaling strategy
related talks
33mCitus & Patroni: The Key to Scalable and Fault-Tolerant PostgreSQL
31mSharding Without the Complexity
32mLoading at Hyperscale: Advanced Data Import Techniques for Citus
52mScaling a Saas application beyond a single Postgres with Citus
28mEfficiently distributing Postgres with Citus – How to choose the right shard key?
44mDistributed Postgres with Citus
28mCitus Cluster: our zero-downtime migration story
25mSelf Hosted Citus using Cloud Computing