Dos and Don’ts: Designing an XDS Backend for the LinkedIn Scale
summary
LinkedIn implements a custom xDS control plane that serves rest.li, Envoy and gRPC clients, migrates registration from Zookeeper to Kafka, and open‑sources the control plane. Rollout reveals a 10x traffic skew caused by DNS ordering and RFC 3484 IPv6 address sorting under the default pick_first; switching to round_robin then exhausts memory because each client eagerly opens connections to every target, filling TLS buffers. Interim fixes include a server banlist that kills reconnecting clients and dynamic DNS removal while shuffled pick_first and connection‑reuse changes roll out.
AI-written summary · about
chapters (14)
- 0:00LinkedIn’s XDS control plane intro
- 1:05Why LinkedIn built XDS control plane
- 3:17ZooKeeper to Kafka migration
- 4:00Traffic issues with gRPC pick-first
- 5:24RFC 3484 IPv6 skew
- 7:12Round-robin gRPC memory issue
- 9:40Banning clients for XDS streams
- 11:49gRPC shuffled pick-first solution
- 12:52Dynamic DNS removal fix
- 13:55Traffic distribution improvements
- 14:16Dero control plane optimizations
- 15:19Zero-copy gRPC buffer reuse
- 16:03Closing remarks
- 16:49Q&A on XDS design
related talks
23mxDS Support in gRPC
29mDNS: Old Solution for Modern Problems
31mHow We Moved Spotify To a Proxyless gRPC Service Mesh
08mLightning Talk: It’s 10pm, do you know what your proxy is doing? Understanding...
41mLinkedIn's Distributed Firewall
10mEnvoy’s Using 10GB of Memory and It’s All My Fault!
35mIntelligent Load Balancing in Kubernetes
40mImplementing the SPDY Protocol in the LinkedIn Social Network