Improving Reliability by Data-Driven Engineering & Culture Changes
summary
Defines a longitudinal metric called advertiser value lost and decomposes it into instantaneous burn rate, detection latency, and mitigation time to prioritize work; uses a three-layer detection stack (system, centralized outcomes, ecosystem slow-burn monitoring) and invests in testing, ML reliability, and production resilience. Builds an ML control loop for model snapshot validation that cuts ML-related losses from hundreds of millions to less than $10M, and reports overall system value loss improving by more than 50%.
AI-written summary · about
chapters (4)
related talks
33mThe Value of Reliability
35mWhat Failure Taught Me About Building High-Stakes Models
34mBuilding a Culture of Reliability
18mMeasuring Reliability Culture to Optimize Tradeoffs: Perspectives from an...
27mThe Perverse Incentives of Reliability
14mBuilding a Data-Driven Culture: An Anthology of Strategies and Insights
25mMaking smart investments: A framework for maximizing your ROI in technical decisions
37mBetter Reliability Through Observability and Experimentation