Building Petabyte-Scale PostgreSQL Deployments
summary
Replaces ElasticSearch after hitting a 1PB limit, citing long GC pauses and cluster rebalances, and builds Bagger: a Kafka-first ingestion pipeline that writes each JSONB message to two Postgres nodes for redundancy, parses JSONB in a C trigger, and ships a patched Postgres (index key limit raised) to support 48 indexed fields. System reports linear write throughput to 10 PB (≈3 TB/day initially, 30-day retention); read queries hit every node so read scalability is limited, but ingestion design works.
AI-written summary · about
chapters (12)
- 0:00Introduction to Bagger
- 1:15Speaker background and talk overview
- 2:01Elasticsearch limitations at petabyte scale
- 6:25Why PostgreSQL for Bagger
- 9:09Design tradeoffs in distributed systems
- 13:12Open-sourcing the original Bagger
- 15:42Rebuilding Bagger in open source
- 21:42Data flow and architecture
- 27:15Current development status
- 28:52Future plans beyond GA
- 31:53Key tradeoffs in scalability
- 37:41Q&A session
related talks
25mPetabyte-scale Log management in Postgres using Bagger
36mPostgres at any Scale
46mBig databases, many servers, data center or cloud - it's all here!
51mScaling Postgres to the next level at OpenAI
37mPostgres at Any Scale
50mPostgreSQL Scale-Out at inMobi
27mBuilding a PostgreSQL data warehouse
31mBiggish Data With Rails and Postgresql