How and why we lowered our SLO
summary
Team uses Google CRE and GCP with Pivotal Cloud Foundry to define SLIs as periodic probes that request tracker and check for HTTP 200 within X ms, build graphs/observable dashboards, and enforce a policy that halt deployments when error budget is exceeded until 10 minutes of budget returns. They find the SLO was too strict, so they relaxed SLO from 99.5% (43 min) to 99.7% (127 min), run disaster recovery drills and practice disaster recovery using error budget while killing old versions and iterating probes.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 0:32Who am I
- 2:11Pivotal Tracker
- 3:03Tracker Team
- 3:45Cloud Ops Team
- 4:29Tracker Application
- 11:24Platform Operator
- 11:50Define and build tools
- 12:25Service level objectives
- 14:09Budget
- 15:56Challenges
- 17:03Conflicting Needs
- 17:24Learnings
- 20:28Versioning
- 21:21Spend your budget
- 22:33Ownership of SLO
- 24:05Boundary subtleties
- 25:12Do you need more reliability
- 26:11Too soon
- 27:29Dont do it all
- 28:35Dont forcefeed
- 29:47Blogs
- 30:13Summary
related talks
47mCultivating Production Excellence
30mTechniques for SLOs and Error Budgets at Scale
09mHow to answer tricky questions the SRE way
23mGoing from 30 to 30 Million SLOs
32m9 Things You Should Do When Starting to Use SLOs
38mImplementing SRE practices: SLI/SLO deep dive
18mSLOs as an Organisational 'Check Engine' Light
29mReducing Trauma in Production