Who Ate My Resources? Trace LLM Performance in Real Time With OTel
summary
OpenTelemetry profiling plus an eBPF profiler traces a FastAPI + GPT2 demo end to end, showing a single forward call using ~39% CPU and one helper function accounting for ~$120/hr of spend; continuous profiles link request traces to exact threads and source lines. The demo exports to Grafana dashboards via the OTEL collector and reduces inferred infra cost by ~20%. OpenTelemetry SDK + eBPF profiler, FastAPI + transformers (GPT2), OTel collector → Grafana.
AI-written summary · about
chapters (11)
- 0:00Tracing LLM resources with OTel
- 1:03LLM application challenges
- 3:31What is OpenTelemetry
- 5:18OTel architecture and data flow
- 7:06Why profiling matters
- 9:15Profiling adoption in OTel
- 11:19Demo setup
- 17:26Dashboards and token usage
- 19:57Traces to profiling in OTel
- 22:49Cost savings insights
- 24:15LLM resource optimization best practices
related talks
25mOptimizing LLM Efficiency One Trace at a Time on Kubernetes
25mDebugging Slow Infrastructure Runs With OpenTelemetry in OpenTofu
15mTuning OTel Collector Performance Through Profiling
26mPlug-and-Play Observability: OTel Auto-Instrumentation for LLMs in OpenSearch
34mOpenTelemetry for Dummies: Instrumenting Go Apps
25mPerfect Match: Correlating Continuous Profiling with Distributed Tracing for Stronger Observability
24mWhy Does Continuous Profiling Matter to Developers?
32mDistributed Tracing in Action: Our Journey with OpenTelemetry