SysAdmin or Sherlock?
summary
Six years of OpenShift support yields concrete diagnostics and fixes: use timestamps and UTC to correlate logs, add aggregator logging and OpenTelemetry metrics so missing app metrics don't leave you blind, and run load tests plus kube-linter to catch bad deployments before they scale. A real cluster showed slow kubectl, stuck terminating projects and a kube‑apiserver using lots of RAM due to 37,000 error messages from a bad CRD field; deleting the dummy value immediately restores normal behavior.
AI-written summary · about
chapters (11)
- 0:00Troubleshooting Kubernetes clusters
- 3:15Detective mindset for outages
- 6:23Structured notes for outages
- 10:47Application layer issue
- 13:20RCA and prevention strategies
- 17:15Kibana for metrics demo
- 18:46Pod capabilities and QoS
- 21:18Cluster performance issues
- 24:11API server overload root cause
- 26:48Monitoring cluster metrics
- 30:22Q&A
related talks
32mSurviving From Endless Issues Coming From 7K+ Kubernetes Clusters
27mRethinking Observability for Off-the-Shelf Tools
12mKeynote: From Outage To Observability: Lessons From a Kubernetes...
35mThriving With Kubernetes On-Call: Best Practices & Lessons Learned
28mImprove Monitoring and Observability for Kubernetes with OSS Tools
18mK8s Troubleshooting Demystified: Five Best Practices
34mReading 100+ Kubernetes Post-Mortems - lessons learned
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)