Debugging at Scale in Production with kubectl debug
summary
Claim: use kubectl debug and ephemeral containers to inspect crashed pods without altering originals, and combine port-forward JDWP to localhost for secure remote breakpoints. Demonstrations show IDE integration (IntelliJ) with developer-observability features: nonblocking snapshots that don't stop threads, conditional logs and metrics, tagging thousands of processes, and replayable captures. Cautions include jdwp security risks, breakpoints freezing production, observability cost (logs ~30% cloud spend) and human-caused breaches (~60%), and that tooling must enforce read-only safety and rate limits.
AI-written summary · about
chapters (14)
- 0:00Debugging at scale intro
- 2:18Debugging is everyone's problem
- 3:44kubectl exec challenges
- 5:30kubectl debug and ephemeral containers
- 7:20Security risks of remote debugging
- 9:25Remote debugging with port forwarding
- 11:40JDWP impractical for production
- 13:56Kubernetes debugging complexity
- 15:00Observability as debugging
- 16:45Developer observability integration
- 17:50Demo: IntelliJ IDEA observability tools
- 21:19Snapshots and conditional breakpoints
- 23:50User-specific debugging with snapshots
- 24:54Closing remarks and Q&A
related talks
34mDebugging at Scale in Production - Deep into Your Containers with Kubectl Debug
34mChaos Debugging: Finding and Fixing Microservice Weak Points
33mDebugging & Profiling Spinnaker Applications Live!
45mDebugging Production: No Re-deploys, No Downtime, No Risk
13mKeynote: Debugging in Prod Without (Lots Of) Pain
43mDebugging and Troubleshooting Microservices
35mCrashLoopBackoff, Pending, FailedMount and Friends: Debugging Common Kubernetes Cluster
43mA Guide to Observability for Your (Java) Apps in K8s