Cattle Not Pets, but Don't Delete It Until Investigated
summary
Proposal uses MachineHealthCheck External Remediation to fence failing nodes instead of deleting them, automating a sequence that powers the node off via cloud API, injects an NMI to capture a kdump to disk, and applies the non-graceful-shutdown taint so pods fail over safely. Implementation idea ties an external remediation operator to fence_kdump (or cloud power fencing) so failover happens quickly while preserving node disks and memory dumps for root‑cause investigation; presented as a viable prototype workflow.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:50Pets versus cattle analogy
- 5:31Investigating failures in cattle
- 7:21Failover issues with volumes
- 10:21Safe failover requirements
- 12:55Ideal failover without deletion
- 14:31Fencing in Kubernetes
- 16:49Machine Health Check
- 18:40Non-graceful shutdown
- 21:58Safe failover goals and challenges
- 22:38External remediation for fencing
- 24:24Case study: Power fencing with taint
- 26:35Preserving memory state with Kdump
- 27:43Fence Kdump reliability
- 30:18Combining features for investigation
- 32:05Challenges and solutions summary
- 33:55Q&A
related talks
37mHow To Handle Node Shutdown In Kubernetes
23mMedik8s - The Medical Toolkit For Your Kubernetes Cluster
27mHow Our Security Requirements Turned Us into Accidental Chaos Engineers
29mFrom pets to cattle, the way of Kubernetes
34mReading 100+ Kubernetes Post-Mortems - lessons learned
33mNot so graceful: K8s node failure and stateful workloads
30mPets, Cattle, Chickens, Humans: A strange Kubernetes love-letter
31mHow the OOM-Killer Deleted My Namespace, and Other Kubernetes Tales