When the Logs Just Don’t Cut It: Root-Causing Incidents Without Re-Deploying Prod
Talkat KubeCon + CloudNativeCon North America 2022 · on Friday, 2022-10-28· at 14:00· Portside Ballroom (Room 260)·
summary
Production pod is crash‑looping on a money validity panic; logs don’t show the bad value. The demo attaches a bpftrace u-probe to the running binary (no redeploy), captures function arguments showing the 99‑unit sticker enters as units=0 nanos=−10000000 (negative ten million), and reveals a buggy post‑sum validity check that increments units incorrectly. Pixie automates deployment of the Pixie daemonset across nodes and the probe runs in-place within minutes, producing the root cause without restarting prod.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 0:21Online Boutique incident
- 3:52Risks of prod logs
- 4:12BPF Trace and eBPF
- 6:44BPF Trace in Kubernetes
- 7:46Demo: BPF Trace logs in prod
- 10:57Root-causing with logs
- 11:39BPF Trace use cases
- 12:23eBPF ecosystem
- 13:03Q&A
- 13:45Pixie security needs
- 15:20Debug symbols in builds
- 16:02BPF Trace with Java
- 17:10Live BPF Trace demo
- 20:33BPF Trace probe duration
- 21:37Deploying Pixie
- 22:40Pixie security benefits
- 23:01Observing running pods
- 23:43External symbol stores
- 24:50Pixie access control
- 26:27Future symbol storage
- 27:29Targeting pods with BPF
- 28:41BPF in pipelines vs prod
- 30:12Closing remarks
related talks
13mKeynote: Debugging in Prod Without (Lots Of) Pain
29mReproducing Production Issues in your CI Pipeline Using eBPF
10mOptimizing incident response
28mBpftrace Meets Pixie: Dynamic Monitoring of K8s Clusters Unleashed
47mDid My Deploy Degrade Production?
36mDude, You Forgot the Feedback: How Your Open Loop Control...
25mDebugging at Scale in Production with kubectl debug
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures