The Brave Little Database!
summary
A small production database runs into a RAM exhaustion crash, reboots into read‑only mode, and accumulates a huge backlog of delayed jobs and indexes that slow writes for minutes. Recovery includes setting up a replica with row‑based replication and Orchestrator automatic failover to promote a healthy node, plus adding ProxySQL multiplexing and reader groups so reads hit replicas and writes stay on primary. The talk shows the crash cause, the exact tools and configs used, and that the fixes actually restore throughput.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:17First challenge
- 2:59Traffic spike and crash
- 4:25Failure recovery
- 5:28Creating a twin
- 6:31Automating failovers
- 7:32Scaling reads
- 8:35Epilogue and new issue
related talks
59mWhy Databases Cry At Night
24mDatabases beyond the tutorials
58mWhy Databases Cry At Night
47mPerfecting business continuity for Azure SQL DB
51mDatabase Disasters and How to Find Them
22mLessons from Automatic Incident Resolution for a Million Databases
54mSoaring to New Heights with Accelerated Database Recovery
44mConfessions of a DBA: worst and best things I've done in production