Keynote: Anatomy of a Production Kubernetes Outage
summary
A production Kubernetes outage lasts 1h21m after a ledger deployment triggers Linkerd to route to stale pod IPs; Linkerd crashes with null pointer exceptions when it reads an empty EndpointSet created by a service scaled to zero. Root causes combine a gRPC client bug in etcd/API server, a version incompatibility between Kubernetes and Linkerd, and harmful human action restarting Linkerd pods. Built defenses include an edge fallback that sustains most card payments, and postmortem reveals gaps in monitoring and lack of chaos testing.
AI-written summary · about
chapters (5)
related talks
12mKeynote: From Outage To Observability: Lessons From a Kubernetes...
37mLinkerd End User Panel: Case Studies from Production
20mCreating Chaos in the University with Linkerd and Chaos Mesh
09mLightning Talk: How to Avoid a Kubernetes Doom Loop
41mKubernetes the Very Hard Way
05mReading 100+ Kubernetes Post-Mortems
33mThe Gotchas of Zero-Downtime Traffic /w Kubernetes
33mWhat We Learned from Reading 100+ Kubernetes Post-Mortems