Stories from the Playbook
summary
Describes how GKE SREs keep hosted masters healthy across 15 regions with a target 99.5% SLO, using per-cluster dashboards, fleet aggregation, probers and a monitoring server that can trigger automatic master recreation or resize masters; releases ship weekly with a weekly four-day rollout and a sliding window of three minor Kubernetes versions. Cites real incidents: Docker OS image regressions fixed by rollback and flag change, corrupted image in Container Registry purged to restore cluster creation, and expedited patched releases for dnsmasq vulnerabilities.
AI-written summary · about
chapters (13)
- 0:00Intro
- 2:54GKE in Numbers
- 4:36Playbooks?
- 6:08Observability
- 7:30How do we monitor clusters?
- 9:25What could possibly go wrong?
- 12:53Playbook Example: etod debugging
- 16:22Common Procedures
- 19:31Incident Management
- 21:08Docker Live Restore
- 23:15Image Corruption
- 26:34dnsmasq vulnerabilities
- 28:56What are we currently working on?
related talks
49mSite Reliability Engineering with Kubernetes
42mMastering Kubernetes: Challenges, Strategies, and Best Practices
36mKubernetes with a single master
33mKubernetes Security: Horror Stories to Help Secure Your Cluster
21mSelf-Healing Kubernetes
36mUnderstanding Scalability and Performance in the Kubernetes Master
34mReading 100+ Kubernetes Post-Mortems - lessons learned
25mAn SLO-Driven Approach to Enhance Kubernetes Cluster Reliability