Loading at Hyperscale: Advanced Data Import Techniques for Citus
summary
Citus parallelizes writes so COPY at 500k/sec becomes realistic while microbatching (100–1,000 rows per COPY) commonly reaches ~100k inserts/sec; index juggling (disable/rebuild indexes) often yields 2–3x faster loads for large replace operations; massive parallel COPY needs file-chunking and careful coordinator CPU sizing; direct worker writes are possible but fragile (recommended only at ~100+ TB), require correct Postgres integer hash reproduction, colocation checks via pg_dist_partition, and tooling like pgloader and dstat.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:59Single row inserts, upserts, & \COPY
- 5:46Microbatched & parallelized \COPY
- 10:18Index juggling
- 14:01Data ordering with case study
- 17:08Distributed triggers
- 22:38Direct worker loading
related talks
16mOnline Scale-Out and Cloud Data Ingestion
48mOptimal Data Loads
46mETL Confessions
25mData-intensive PostgreSQL: Three ways to scale
37mPostgres at Any Scale
18mMaking Postgres inserts faster on Azure
31mMove Less, Move Faster: Speeding Up Citus Cluster Scaling
24mLessons Learned from benchmarking and profiling distributed PostgreSQL