Walk-through: Debugging an RBAC Problem in Istio (But Without the Swearing)
Talkat KubeCon + CloudNativeCon North America 2019 · on Thursday, 2019-11-21· at 11:50· Room 6F - San Diego Convention Center, Upper Level·
summary
Reproduces a real Istio RBAC outage and shows exact steps to find the root cause: requests get 403 because Envoy sidecar denies them when caller identity is empty. Uses Envoy access logs, the on guard access-log parser, live packet captures (tcpdump/pcap via a kubectl plugin), istioctl/ pilot and proxy-status checks, and toggling permissive RBAC mode. Builds experiments that change one thing at a time; finds mTLS identity was missing so RBAC comparisons fail, fixes by restoring mTLS / correct config and traffic returns.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:22Turning on RBAC
- 3:02Debugging Principles
- 4:30Access Logs
- 7:09OnGuard
- 11:03Control
- 11:20Permissive Mode
- 14:45Mental Model
- 19:31Pipeline
- 23:30Guess Check
- 29:03Conclusion
related talks
33mObserving and Troubleshooting your Microservices with Istio
36mIstio, we have a problem! Understanding and fixing bugs with a service-mesh
45mIstio to the Rescue
35mReliable Application Roll Out and Operations with Istio [B]
47mIstio, We Have a Problem! Understanding and Fixing Service Mesh Bugs
36mIstio: Real world scenarios
46mService Mush: Debugging Istio Deployments
24mUnraveling Istio: A Tale of Scale, Challenges and Triumphs at DevRev