Cultivating Production Excellence
summary
Argues that production excellence requires measurable SLOs, observable telemetry, collaboration, and risk-driven prioritization; recommends defining SLI thresholds like 99.9% SLO over 30 days with clear eligible events, using the error budget = allowed failures to gate alerts and experiments. Cites a real incident: a 2% brownout for ~20 minutes missed by black‑box probes. Describes a working outcome: a small team (nine engineers) runs ~2,000 deployments/year with <2 catastrophic failures, rollback <10 minutes, fix‑forward ~20 minutes.
AI-written summary · about
chapters (36)
- 0:00Intro
- 1:41Production is increasingly complex.
- 3:03We're adding complexity all the time.
- 6:10Our strategies need to evolve.
- 6:42When we order the alphabet soup...
- 8:06Noisy alerts. Grumpy engineers.
- 8:39Walls of meaningless dashboards.
- 11:54Tools aren't magical.
- 12:29Invest in people, culture, & process.
- 14:22Eliminate (unnecessary) complexity.
- 14:47Our systems are always failing.
- 15:50We need Service Level Indicators
- 17:41What threshold buckets events?
- 18:40Set a target Service Level Objective.
- 23:24Data-driven business decisions.
- 26:11Failure modes can't be predicted.
- 26:33Support debugging novel cases. In production.
- 27:13Allow forming & testing hypotheses.
- 28:49Can you explain the variance?
- 31:01Observability isn't just the data.
- 32:31Debugging is not a solo activity.
- 32:59Debugging is for everyone.
- 33:27Collaboration is interpersonal.
- 33:42Lean on your team.
- 35:13Fix hero culture. Share knowledge.
- 35:44Use the same platforms & tools.
- 36:03Reward curiosity and teamwork.
- 36:37Risk analysis helps us plan.
- 37:08Quantify risks by frequency & impact.
- 40:02And prioritize completing the work.
- 40:16Don't waste time chrome polishing.
- 40:35Lack of observability is systemic risk.
- 41:12So is lack of collaboration.
- 43:41A dozen engineers build Honeycomb.
- 43:54We make systems humane to run
- 44:05Yes, we deploy on Fridays.
related talks
29mCultivating Production Excellence
58mCultivating Production Excellence
29mReducing Trauma in Production
51mShrinking Production Incidents
39mShifting Left on Production Excellence with Observability
25mHow to detangle and understand production
03mCultivating production excellence
30mMetrics for good, not evil