What Happens When Something Goes Wrong? On Kubernetes Reliability
summary
Kubernetes auto-recovery is concrete: kubelet restarts crashed containers with exponential backoff and supports liveness/readiness probes (etcd needed a longer initialDelaySeconds in GKE masters). Node-level tools include node problem detector reports kernel issues and the node controller heartbeats (10s/40s thresholds) plus eviction throttling (default one node per 10s, slow one per 100s). Alpha taint-based eviction lets pods set tolerationSeconds so critical or disk-local workloads avoid premature eviction; GKE adds automated master/node repair and upgrades that recreate VMs when components go unhealthy.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:55What will we be speaking about?
- 3:01Pod Replication
- 3:38Container Lifecycle management
- 7:06Probe Types
- 7:36Configuring Liveness Probes (GA) livenessProbe: httpGet
- 9:13If Pod itself is not broken then what? Environment?
- 11:07Environmental problems
- 11:45How is the environment getting foed?
- 15:53Diagnosis Daemons
- 16:37Node Problem Detector
- 17:58Hardware or software?
- 18:27How does Kubernetes check this?
- 19:28How does the system handle unresponsive Nodes?
- 20:51What happens when everything is very wrong?
- 24:46Configurability
- 26:34Taints and Toleration example (alpha)
- 28:58Control plane issues
- 32:22Nothing is perfect
- 32:49What does GKE do for you when things go wrong?
- 34:08Thank you for listening
related talks
35mKubernetes, Lessons Learnt
39m15 KUBERNETES FAILURE POINTS YOU SHOULD WATCH
34mReading 100+ Kubernetes Post-Mortems - lessons learned
34mBecoming an Expert of Diagnosing a Kubernetes Control Plane Problem
10mOptimizing incident response
09mKeynote: Reliability and Performance Testing for Kubernetes Operators
29mKubernetes Failure Stories and How to Crash Your Clusters
42mMaking Resilient, Self-Adaptive, And Autonomous Kubernetes Clusters