Beyond Generic Spans: Distributed Tracing for Actio...
summary
Implemented end-to-end tracing for llm-d with manual OpenTelemetry instrumentation that adds targeted spans (gateway/EPP scoring, KV cache lookups, PD proxy transfers, vLLM request phases) across multi-node inference. Traces expose when routing to a warm KV cache yields measurable time-to-first-token improvement, when prefill/decode disaggregation is chosen vs unified processing, and where MoE hotspots occur; instrumentation lands in vLLM V2 and is enabled in deployments with two helm flags, and it works in production.
AI-written summary · about
chapters (8)
- 0:00Tracing in LMD
- 1:07Auto instrumentation limits
- 2:12Inference pipeline components
- 3:21Prefix cache routing
- 6:15Decode disaggregation tracing
- 8:46Tracing in VLLM setup
- 9:48End-to-end visibility benefits
- 10:33Conclusion and demo
related talks
32mFrom Traces To Action: Auto-Instrumenting LLMs for Observability...
23mObservability for LLMs: Understanding What’s Happening Under the Hood
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
18mWhat did AI just do? Making LLMs Observable with OpenTelemetry
26mOpenTracing Isn't Just Tracing: Measure Twice, Instrument Once
34mReduce Complexity with Distributing Tracing
32mEnd-to-End Observability for LLM Inference: From Token To GPU
30mBeyond OpenTracing