Practical Guide to Migrating +100 Large PostgreSQL to the Cloud
summary
Migrates more than 100 production PostgreSQL databases (≈250 including dev/prod copies) from VMs to Google Cloud SQL using pglogical and automated Bash+YAML orchestration; automation runs ~20 sequential steps in parallel waves. Finds initial data copy for 40–60GB tables can take 48–70 hours unless non‑PK indexes are dropped and rebuilt on target; WAL accumulation, hundreds of replication slots, memory limits for subscribers, partitioning→crash and inheritance/column‑order incompatibilities require bespoke fixes. Validation uses row counts and 500‑row hash samples; overall approach works with careful capacity planning and cost tuning.
AI-written summary · about
chapters (9)
- 0:00Introduction and project overview
- 4:23Proof of concept and planning
- 10:25Choosing migration tools
- 13:39Initial data copy challenges
- 18:20Capacity planning and automation
- 25:23Validation and troubleshooting
- 31:17Common migration problems
- 45:25Key lessons and recommendations
- 47:59Q&A and conclusion
related talks
59mLarge Scale MySQL Migration to PostgreSQL
46mBig databases, many servers, data center or cloud - it's all here!
50mFrom VMs to Cloud-Native PostgreSQL in Kubernetes...
26mMastering Large-Scale PostgreSQL Migration
49mLessons from Migrating a 10 TB PostgreSQL Cluster to AWS
42mMigrating a database over 10TB from DB2 to PostgreSQL with zero downtime
29mTales from the Field - Oracle to PostgreSQL migrations
37mDatabase Modernization at Cloud Scale: Minimizing Risk and Disruption