Envoy on Fire: A Practical Look at Debugging a Service Mesh
Talkat KubeCon + CloudNativeCon North America 2019 · on Thursday, 2019-11-21· at 17:20· Ballroom Sec 20AB - San Diego Convention Center, Upper Level·
summary
Shows how to triage Envoy with concrete tools: curl the admin config dump to see bootstrap and xDS provenance, enable trace-level logs to find DNS or xDS rejections, and recognize Envoy’s “last known good” behavior that lets bad xDS surface only on scale-up. Demonstrates diagnosing user 404s via the NR response flag, building egress logging with filters and sampling, and shipping logs over gRPC to an access log service. Recommends monitoring cluster stats such as upstream_request_pending_active vs total, membership and health checks, and surfacing them in incident dashboards.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:52Envoy Configuration - The Bigger Picture
- 3:36Envoy Tools at Our Disposal
- 5:12Envoy Configuration - Scenario 1
- 7:11Envoy Configuration - Scenario 2
- 13:08Selective Logging of Traffic to an External Service
- 14:32Configure Listeners
- 16:53Sample 1% External Traffic Logs
- 17:44Log only Errors for External Traffic
- 19:34Log Slow Calls for Auth'd External Traffic
- 20:04Structured Access Log Events via GRPC
- 22:35Go Control Plane - Access Log Server
- 24:01Logs Takeaways
- 25:55Taxonomy of Stats - Top Level Metrics
- 28:16Process Stats
- 30:15Useful Stats - Service Discovery
related talks
09mLightning Talk: The Hardest Part of Operating a Service Mesh: Envoy Proxy
08mConnecting the (Proxy) Dots: A Beginners Guide to Reading Envoy Debug Logs
39mNetwork Observability with Envoy
01h 25mCloud-Native Building Blocks: An Interactive Envoy Proxy Workshop
01h 32mHands-on Workshop: Envoy Metrics Demystified Workshop
33mIntro: Envoy
11mxDSnap: A Lightweight Tool to Debug Envoy and Consul Dataplanes with Confidence
20mKeynote: Service Meshes and Observability