Keeping High-Traffic Systems Running When Things Go Wrong
summary
Specifics: Postgres queries slow from ~15ms to ~150ms, one user action triggering six internal calls turns 100ms pieces into 600ms end-to-end, P99 spikes to ~5s while P50 stays low, connection pools grow to ~50 and block new requests. Practical mitigations are concrete: limit concurrency and DB queries, short layered timeouts that cancel work (client 2–3s, service 500ms–1.5s), circuit breakers, returning 429s, graceful degradation, intentional load shedding, and tracing with correlation ids. Watch latency percentiles, retry storm, connection pool to 50.
AI-written summary · about
chapters (7)
- 0:00Introduction and Background
- 0:49Understanding System Failures
- 2:21Real-World Example of System Failure
- 8:23Strategies for Stabilizing Systems
- 12:24Monitoring and Incident Response
- 16:29Designing for Reliability
- 18:21Conclusion and Final Thoughts
related talks
49mHELP!!1! The database is slow!!
46mHow we made PG Fitter, Happier, More Productive
47mResolving performance issues in practice
28mPlanning next moves: improving performance when half your stack is someone else's problem
29mReady, steady, crash!
25mReady, Steady, Crash!
36mMake It Fast
09mLightning talk: The 6 Characters That Could Bring Down Your App