Museum of Weird Bugs: Our Favorites From 8 Years of Service Mesh Debugging

on YouTube ↗30m

" summary

Eight years of Linkerd debugging yields concrete fixes: a CRD/schema mismatch causes an infinite patch loop and 780 MB leak because a generated Rust struct includes a port field Kubernetes drops; fix is to align the CRD. Another bug shows a gRPC/HTTP2 flow-control deadlock where blocking stream.send in a Go destination controller halts all informer callbacks; fix is a per-listener queue and stream termination for lagging clients. Tools: Kubernetes API, CRDs/Gateway API, client-go informers, gRPC over HTTP2, Rust and Go.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 1:26Linkerd architecture overview
3. 8:03CRD status update memory leak
4. 15:25Stale routing and deadlock
5. 27:31Proxy memory leak
6. 28:13Q&A

" related talks

• 28mThere's a Bug in My Service Mesh! What Do You Do When the Mesh is At Fault?Ana Calin, Risha Mars · KubeCon + CloudNativeCon North America 2019
• 33mLove your bugsAllison Kaptur · PyCon 2018
• 34mChaos Debugging: Finding and Fixing Microservice Weak PointsIdit Levine, Mitch Kelley · KubeCon + CloudNativeCon Europe 2019
• 34mReading 100+ Kubernetes Post-Mortems - lessons learnedNoaa Barki, Shimon Tolts · Conf42 Site Reliability Engineering 2021
• 29mObservability and Debugging Cloud-Native Apps using Linkerd & ConduitFranziska von der Goltz · KubeCon + CloudNativeCon Europe 2018
• 36mIstio, we have a problem! Understanding and fixing bugs with a service-meshDavid Gageot · DevFest Nantes 2018
• 20mOverview and State of LinkerdTarun Pothulapati, William Morgan · KubeCon + CloudNativeCon North America 2020 Virtual
• 35mOverview And State Of LinkerdAlex Leong · KubeCon + CloudNativeCon North America 2022