Production Testing through Monitoring
summary
Argues that testing alone fails at scale and actual production monitoring must prove the business is working not just servers. Shows concrete checks: trace message delivery rate, watch customer-facing flows (login, purchases, email bounces) and compare them to hard counters like billion emails per month or spikes from 300,000+ users. Recommends instrumenting and alerting on business-level metrics, using existing services (eg S3 and messaging systems) to detect real failures.
AI-written summary · about
chapters (17)
- 0:00Start
- 1:09a discussion on trade-offs between testing and monitoring and how to apply them in a devops lifecycle
- 2:28testing is overrated
- 3:09resilience testing
- 3:43testing can give a false sense of security
- 3:56testing is deterministic
- 6:37the Corrupted Blood incident
- 10:30100% code coverage is not the goal
- 10:55the goal of testing is to win the confidence game
- 11:32we want to be reasonably confident that the code we are going to push will not break production
- 13:32enter monitoring
- 14:05systems are complex external dependency worry proactive is better than reactive
- 16:18integration points performance user behavior
- 20:26most people use monitoring to focus on the wrong things
- 22:30top-down approach understand business define baseline correlate data
- 27:16add observability to everything
- 27:35alert only on actionable emergencies
related talks
28mProduction Testing Through Monitoring
30mReducing Chaos in Production: Testing vs. Monitoring
58mObserving Applications in Production
36mIt's not in Production Unless it's Monitored
34mTesting in Production
50mFrom Load Testing to Reliability Engineering: Making Performance Testing Predict Production Behavior
28mDo You Know If Your Service Is Working Properly?...
19mDo more with less: Unit and integration testing is broken