Petabyte-scale Log management in Postgres using Bagger
summary
Bagger manages petabyte-scale logs by ingesting JSON into Kafka and sharding writes into Postgres pairs, replacing a previous ElasticSearch setup that failed at ~1 PB. It claims linear write scalability, measures Postgres write amplification around 4x versus Cassandra 8x, and uses a C-language trigger plus Perl tooling for ingest and topology control. Metadata servers, node control and CD replication exist, the C trigger is in final development and a streaming query proxy is pending; experiments include ZFS compression and compiling Postgres with up to 32KB blocks.
AI-written summary · about
chapters (13)
- 0:00Introduction and background
- 1:27Elasticsearch limitations at petabyte scale
- 2:52Why Postgres for log management
- 4:17Open-sourcing the predecessor project
- 5:40Design choices for Bagger
- 7:07Data flow architecture
- 8:55Current project status
- 10:20Future improvements and trade-offs
- 13:09Key lessons in distributed data
- 15:36Specialization vs generalization
- 16:04Q&A: Production use and alternatives
- 18:32Q&A: Columnar databases and compression
- 21:59Q&A: Query proxy workloads
related talks
48mBuilding Petabyte-Scale PostgreSQL Deployments
36mPostgres at any Scale
19mA New Approach to Logging as a Stack: Fluent Bit + PostgreSQL (FPS)
34mAnalytics and AI with PostgreSQL - Mission Possible!
51mScaling Postgres to the next level at OpenAI
20mWrites, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™
31mBiggish Data With Rails and Postgresql
27mBuilding a PostgreSQL data warehouse