Online Scale-Out and Cloud Data Ingestion

on YouTube ↗16m

" summary

Uses Azure Blob Storage and the Citus extension for Azure Cosmos DB for PostgreSQL to ingest a compressed CSV directly into Postgres and loads ~3,000,000 rows via COPY/BlobGet. Shows Create distributed table concurrently (Citus 11.11) that avoids blocking writes, and moves a hot tenant shard (shard 102093 ~78MB → new shard 102110 ~74MB) online while inserts continue. Introduces a background rebalancer job that evens worker sizes (69/151MB → 73/146MB) without holding a client connection.

AI-written summary · about

" chapters (7)

1. 0:00Scale-out Citus clusters
2. 1:09Azure Blob Storage ingestion
3. 4:44Distributing tables without blocking writes
4. 7:11Online table distribution
5. 9:42Detecting skewed data distribution
6. 11:52Moving large tenants non-blocking
7. 13:45Rebalancing shards in Citus 11.1

" related talks

• 24mCitus from the Customer eyesSai Krishna Srirampur · Citus Con: An Event for Postgres 2023
• 52mBuilding Distributed OLTP with Azure Cosmos DB for PostgreSQLCharles Feddersen · SQLBits 2023
• 32mAuto scaling Azure Cosmos DB for PostgreSQL with Citus, Grafana & Azure ServerlessLucas Borges · Citus Con: An Event for Postgres 2023
• 31mKEYNOTE: The Distributed PostgreSQL Problem & How Citus Solves itMarco Slot · Citus Con: An Event for Postgres 2023
• 27mMulti-tenant Saas apps made simple on Azure Cosmos DB for PostgreSQLAdam Wolk, Alicja Kucharczyk · Citus Con: An Event for Postgres 2023
• 44mMulti-tenant SaaS apps made simple on Azure Cosmos DB for PostgreSQLAdam Wolk, Alicja Kucharczyk · SQLBits 2023
• 20mLessons learned scaling our SaaS on Postgres to 8+ billion eventsJonathan Denney · Citus Con: An Event for Postgres 2022
• 32m8 keys to the growth of Citus & Distributed PostgreSQLUmur Cubukcu · Citus Con: An Event for Postgres 2022