Securing Production Debugging at Hyperscale
summary
Microsoft debugs a global K8s platform called Cosmic that runs 12,000 clusters, tens of millions of pods and nearly 6 million cores by issuing short-lived SSH certificates with SSH force‑command constraints, JIT SSH servers, and an OS‑agnostic controller that executes commands against pods or nodes with full stdout/stderr logging and per‑command audit trails. An AI orchestrator (COP + agent‑to‑agent) runs read‑only diagnostics across 19 scenario types, averages ~37s responses, and reports ~85–90% root cause accuracy with forced citations and 10‑cycle limits.
AI-written summary · about
chapters (8)
- 0:00Securing production debugging
- 1:50Microsoft Cosmic overview
- 3:18Traditional debugging challenges
- 4:43Cluster Debugging Tool intro
- 8:34AI-driven agent debugging
- 11:21AI agent security
- 13:48CDT and agent future
- 14:29Summary
related talks
34mDebugging at Scale in Production - Deep into Your Containers with Kubectl Debug
24mDebugging at Scale in Production - Deep into your Containers
01h 00mDebug like a pro on Kubernetes
51mTroubleshooting & Debugging Production Microservices in Kubernetes
35mThe Art of Debugging Inside K8s Environment
34mRunning a DevOps style production Docker cluster using the Microsoft platform
01h 32mTutorial: Debug Your Kubernetes Apps
50mTroubleshooting & Debugging Microservices in Kubernetes