101 Ways to “Break and Recover” Kubernetes Cluster
summary
Oath (Yahoo) operates about a dozen on‑prem Kubernetes clusters (7000+ pods per uploader) and recounts concrete outages and fixes: a single mistaken namespace delete removes 100+ Deployments and is mitigated by failover plus a namespace-deletion admission controller; duplicate Ingress hosts cause mixed traffic and are blocked by an ingress-claim validation; a kubelet roll‑out bug triggers cluster‑wide eviction backoff and full‑disruption mode; an etcd upgrade script wipes data and is recovered with etcd snapshot restore; custom tooling includes an ATS‑based custom ATS ingress controller, a Screwdriver templating CI/CD, and Athens auth hooks.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:58Where are we today
- 3:07Architecture
- 6:43Pipeline
- 8:15Namespace
- 8:51Delete
- 9:46cascading impact
- 10:27how to recover
- 12:11single ingress controller
- 12:46copy paste clone
- 13:49ingress claim
- 18:32degrade experience
- 20:42API connection
- 21:27Recover
- 22:23TLS Rotation
- 24:51Multiple Public Keys
- 25:29Ecosystem
- 25:52Cube DNS
- 27:55DNS Mask
- 29:05Kubernetes Releases
- 30:17Container Isolation
- 31:06Node Controller
- 32:23Game Day Testing
- 34:24Open Source
related talks
36m10 More Weird Ways to Blow Up Your Kubernetes
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster
35m10 Weird Ways to Blow Up Your Kubernetes
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
35mKubernetes, Lessons Learnt
45mWhat I Learnt Fixing 50+ Broken Kubernetes Clusters
34mReading 100+ Kubernetes Post-Mortems - lessons learned
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)