Tracing HTTP/2 traffic using gRPC tapping filter with hardware acceleration
summary
Built a streaming gRPC tap sink that pushes connection-level traces off the main thread, adds a ConnectionInfo cache replay so restarted collectors get prior connection metadata, and emits counters for traced traffic; replaces admin-file tapping that bottlenecks the main thread. Overhead measures show <5% throughput loss and ~1–2 ms added latency at peak, and it traces hundreds of thousands to a few million RPS in production. Memory copies use a transparent DSA offload (threshold ≈200–256 KB) to shift large-copy work to PCI DSA hardware.
AI-written summary · about
chapters (17)
- 0:00gRPC tapping filter intro
- 1:25HTTP/2 traffic tracing in 5G
- 3:12Metrics and logs limitations
- 5:45Envoy filter architecture basics
- 8:09Layering Envoy filters
- 9:13Selective tracing with sockets
- 11:19Avoid file/admin syncs
- 13:51gRPC sync solution design
- 16:23Connection replay mechanism
- 17:49HTTP vs connection tapping
- 18:34Tracing stats and overhead
- 19:59Hardware-accelerated memory copies
- 23:07Intel Data Streaming Accelerator
- 26:24DSA performance testing
- 28:34Hardware acceleration future
- 29:35Upstreaming tapping filter status
- 30:19Q&A
related talks
01h 00mSystems @Scale Summer Q&A Week 4
22m"Low-Overhead Observability in High-RPS Servers"
12mStriaTrace: Efficient Tracing and Diagnosis for Online LLM Inference
35mHigh Throughput with Low Resource Usage: A Logging Journey
46mVisualising and tracing requests through a cluster: Intergrating OpenTracing into OpenStack Swift
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
26mPerformance Analysis of Cloud Applications
36mThe black art of network performances