Database Disasters and How to Find Them
summary
Checklist-driven incident response: confirm connectivity, check logs and I/O, and fail over if the instance is truly down. Gives concrete fixes from real incidents: a 178 GB table with 23 indexes causes bloat and a plan flip, resolved by vacuum with index_cleanup=false and rebuilding indexes (vacuum finished in 45 minutes); a 25,000-session multi-xact stall is solved by drop FK and batch inserts and switching to single large transactions; a Django recursion creates 100+ subtransactions, fixed by tail iteration. Uses PGBadger, Performance Insights, XID and bloat monitoring and enforces a single coordinator in Slack-style incident management.
AI-written summary · about
chapters (30)
- 0:00Intro
- 0:41The typical incident response.
- 3:21The basics.
- 5:26Initial data-gathering.
- 9:13Database server hard-down.
- 10:33Capacity-related.
- 12:04Check the logs.
- 13:07What does the application say?
- 16:16Problem distribution.
- 16:48Code-related.
- 18:22Session state.
- 20:14Every unhappy database...
- 20:29Site suddenly fell over.
- 21:24Things that didn't work.
- 21:58The observation that mattered.
- 22:41Plan flip!
- 24:31Shutdown Approacheth.
- 25:39What happened?
- 26:19PostgreSQL 12 to the rescue.
- 27:36Mystery Mayhem.
- 31:43Upgrade Follies.
- 35:40Death by Subtransaction.
- 36:13Diagnosis steps.
- 39:35Collegiate responsibility.
- 41:15One-Slide Crew Resource Management.
- 43:43Communication channel protocol.
- 45:19Your incident reports suck.
- 46:15Human error.
- 46:47The answer is usually better monitoring.
- 47:29If everything's an alert, nothing is.
related talks
51mDatabase Incident Management: When Everything Goes Wrong
38mWhen it All Goes Wrong: Database Incident Response Dos and Don'ts
49mHELP!!1! The database is slow!!
37mWhen it all goes Wrong (with Postgres)
01mAre you an "accidental" DBA?
11mThe Brave Little Database!
21mSQL Horror Stories
42mPostgres: stories that will scare you