How We Discover and Locate k8s Cluster Problems Before Users at Alibaba

on YouTube ↗29m

" summary

Describes Alibaba’s in-house tool KubeProbe and related practices for early detection and localization of problems in large-scale Kubernetes clusters. Covers a universal link-detection approach that simulates broad user behaviors, targeted inspections of cluster metrics to spot current or future risks, system enhancements for faster root-cause analysis, and post-detection actions including checks, self-healing and Chat-Ops.

AI-written summary · about

" chapters (13)

1. 0:00Intro
2. 0:42Agenda
3. 1:20Background
4. 5:34Challenges
5. 7:05Thoughts
6. 11:02Solution
7. 12:08Link Detection
8. 14:46Targeted Inspection
9. 15:23System Implementation
10. 19:44After Problem Discovery
11. 22:58Root Cause Analysis and Post-Processing
12. 25:33Chat-ops
13. 26:47Example Demo

" related talks

• 35mNo More Chaos: Audit and Inspect Kubernetes at ScaleMa Jinjing, Jie Chen · KubeCon + CloudNativeCon + Open Source Summit China 2019
• 35mNo More Chaos: Audit and Inspect Kubernetes at ScaleMa Jinjing, Jie Chen · KubeCon + CloudNativeCon China 2019
• 18mK8s Troubleshooting Demystified: Five Best PracticesMaria Ashby · Conf42 Kube Native 2023
• 37mProactive Kubernetes Security: Anomaly Detection and Runtime Alertin...Amit Schendel, Remi Minnebo · CloudNativeSecurityCon North America 2024
• 32mSurviving From Endless Issues Coming From 7K+ Kubernetes ClustersWanhae Lee, Seok-yong Hong · KubeCon + CloudNativeCon North America 2022
• 33mAm I Using It Right? Checking Best Practices on Live Kubernetes ClustersAdam Wolfe Gordon, Varsha Varadarajan · KubeCon + CloudNativeCon North America 2019
• 40mThe Good, the Bad, and the Ugly of Managing Production Systems with KubernetesXin Zhang · KubeCon + CloudNativeCon North America 2016
• 29mProduction Cluster Monitoring and Remediation for High ReliabilityShijun Qian, YingKe Liu · KubeCon + CloudNativeCon China 2018