Weighing a Cloud: Measuring Your Kubernetes Clusters
Talkat KubeCon + CloudNativeCon North America 2019 · on Tuesday, 2019-11-19· at 15:20· Room 30ABCDE - San Diego Convention Center, Upper Level·
summary
Practical checklist for cluster debugging: collect Prometheus metrics and query with PromQL, inspect kube-apiserver logs and health endpoints, and read etcd data with etcdctl or augur on the bolt DB when the API is down. Detect offline nodes with the Prometheus scrape up metric and slow kubelets by high scrape_duration (>~2s). Watch etcd object counts — we’ve seen etcd object count = 1,000,000 and 1,200 updates to one object blow past a 2GB default DB and cause crashloops. Note metric fixes (API latency bucketing) land in Kubernetes 1.14.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:59What we are going to cover
- 2:11Prometheus Data Model
- 5:15Using Prometheus Metrics
- 6:23Differential Diagnoses
- 7:03Full-Stack Debugging
- 8:50Control Plane
- 10:07Master Kubelet
- 11:01Introspecting Components
- 11:50KAS (Kube-apiserver)
- 18:10Kubelet Example
- 24:18Etcd Example
- 30:05Another kube-apiserver example
- 32:25Handling metric issues
- 33:21Metrics Overhaul (1.14)
- 34:50Metric Stability Framework
- 35:19Stability Metadata
related talks
30mOperating within Normal Parameters: Monitoring Kubernetes
19mKeynote: Monitoring Kubernetes Clusters with Prometheus
39m15 KUBERNETES FAILURE POINTS YOU SHOULD WATCH
23mLearning Kubernetes Through the Lens of Metrics
32mSurviving From Endless Issues Coming From 7K+ Kubernetes Clusters
11mKubernetes Observability
29mWhen Kubernetes Misbehaves – Pitfalls and Lessons Learned
34mBecoming an Expert of Diagnosing a Kubernetes Control Plane Problem