Not so graceful: K8s node failure and stateful workloads
summary
Demo uses BusyBox containers on a 3‑node cluster with 4 pods (2 deployments, 2 statefulsets) and PVCs on NFS/iSCSI via CSI (NetApp Trident mentioned). Stopping kubelet shows Kubernetes marks a node not ready in ~33–40s, deployments immediately spawn replacements while RWO volumes block with a multi-attach error, and RWX volumes let two pods write the same file causing data corruption. StatefulSet prevents reschedule until termination. Forced deletes trigger Kubernetes to assume unmount after a 6‑minute max wait, allowing risky remounts.
AI-written summary · about
chapters (8)
- 0:00Intro
- 8:37Demo
- 11:59Node failure
- 14:07Node status updates
- 18:17K8s deployment
- 23:55Unknown duration timer
- 28:51Lessons learned
- 29:44Questions
related talks
42mImproving Availability for Stateful Applications in Kubernetes
27mØ Kubernetes Failover Improvement: Non-Graceful Node Shutdown
29mStateful Workloads in Kubernetes: A Deep Dive
37mHow To Handle Node Shutdown In Kubernetes
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
01h 19mTutorial: Back to Basics: Hands-On Deployment of Stateful Workloads on Kubernetes
32mYour Pods can be stateless, your VMs cannot! How to manage state in k8s
29mKubernetes Workload Resiliency in Action: Beyond Basics