Lessons learned: Migrating from AWS-Hosted PostgreSQL RDS to Self-Hosted Citus

on YouTube ↗24m

" summary

Jellyfish moves from AWS RDS to self-hosted Citus to cut a roughly 40% RDS premium, build a sharded cluster and actually scale; work includes denormalizing customer_id denormalization into most tables, changing PKs/FKs/unique constraints (billions of rows updated), and adding PG BackRest, PG Bouncer connection pooling and Patron for HA. Django multitenant tools cover ~80% of cases but some WHERE-clause rewrites blow up queries by 100x–1000x; rollout uses a slow multi‑DB migration rather than a big bang.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 2:16Directed Acyclic Graph (DAG) at Jellyfish
3. 4:48Why Citus?
4. 8:16Infrastructure Design Choices
5. 12:03What needs to happen to your database schema
6. 15:58Changes to your codebase
7. 18:43Our Rollout Strategy
8. 20:52Wrap-Up & what we’re excited about

" related talks

• 52mScaling a Saas application beyond a single Postgres with CitusJesse H. Willett · PostgresOpen Silicon Valley 2017
• 20mLessons learned scaling our SaaS on Postgres to 8+ billion eventsJonathan Denney · Citus Con: An Event for Postgres 2022
• 49mLessons from Migrating a 10 TB PostgreSQL Cluster to AWSAleksandr Kukushkin · HighLoad++ Moscow 2018
• 46mBig databases, many servers, data center or cloud - it's all here!Flavio Gurgel · Devoxx France 2021
• 44mDistributed Postgres with CitusWill Leinweber · HighLoad++ 2017
• 28mCitus Cluster: our zero-downtime migration storyMatthieu Dumont, Antoine Hemery · Citus Con: An Event for Postgres 2022
• 36mPostgres at any ScaleCraig Kerstiens · Birmingham on Rails 2020
• 40mPushing past PostgreSQL’s vertical scaling limits - Running Amazon RDSAlisdair Owens, Andrei Dukhounik · PGConf.dev 2024