"Low-Overhead Observability in High-RPS Servers"
summary
Netflix replaces parts of their gateway with the Apollo Router in Rust and uses the tracing crate to instrument high‑RPS services, profiling an initial 10–15% CPU cost down to about 1–2% by using call-site caching, making per-layer filters actual filter structs (a fork fixed a metric-layer bug and recovered ~5–10% CPU), adding sampling and a reload layer for live enabling, using fast properties to target instances and 10% sampling, and validating behavior with tracing-mock tests.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:49Netflix’s Layer 7 Gateway and GraphQL Federation
- 2:25Tracing crate overview
- 4:48Tracing overhead hierarchy
- 7:52Static filtering and caching
- 10:55Debugging filter overhead
- 12:34Production lessons
- 14:30Configuring nfilter
- 16:04Testing with tracing-mock
- 16:57Tracing overhead to 1-2%
- 18:08Dynamic call-site reloading
- 19:20Fine-grained call-site filtering
- 20:53Sampling and fast properties
- 21:19Key takeaways and closing remarks
related talks
01h 00mSystems @Scale Summer Q&A Week 4
55mScaling out operational research with Rust
47mAre we observable yet?
46mAdopting Rust by gracefully oxidising web applications
26mProfiling Mastery: Decode Your Data and Elevate Observability
37mMonitoring Service Architecture and Health with BPF
29mLessons Learned: Porting a Streaming Data Pipeline from Scala to Rust.
36mProfiling in the cloud-native era - Matthias Loibl