Biggish Data With Rails and Postgresql
summary
Database conventions break once tables hit hundreds of millions of rows: Honeybadger runs ~1 TB of error rows and ingests ~2 GB/day, and naive count() or offset/limit pagination stalls (pages ~2000). Practical fixes are buying real hardware with lots of RAM and fast separate disks, using EXPLAIN and switching to range queries not offsets, tuning Linux read‑ahead and Postgres with PG‑tune, using partitioning for fast drops and pgbouncer/streaming replicas and Wall‑E for incremental backups.
AI-written summary · about
chapters (14)
- 0:00Start
- 7:04ORACLE
- 11:39Lots of Disks (DB and OS on separate HDDs)
- 13:54Mo' rows Mo' problems
- 16:10Sorting (is tricky)
- 18:30Increase Read-Ahead
- 19:22Use a modern filesystem
- 20:05Tell PG about all that RAM
- 20:39Vacuum Regularly
- 22:14Too Many DB Connections
- 23:01Too Many Locks
- 25:14Intensive DB Queries
- 25:55Partitioning for deletion & archival
- 27:07Backups take forever
related talks
26mBiggish Data With Rails and Postgresql
36mPostgres at any Scale
48mHow to Tame A Mastodon: Lessons for PostgreSQL at Scale
46mBig Data. Or Just Too Much Data?
48mBuilding Petabyte-Scale PostgreSQL Deployments
30mPuny to Powerful PostgreSQL Rails Apps
52mBulk data processing and PostgreSQL thingy
46mBig databases, many servers, data center or cloud - it's all here!