From Hallucinations To Hardware: Diagnosing LLM Failures
summary
Live demo shows an LLM app that surfaces both entertaining hallucinations and real GPU faults and traces them from user request to infrastructure using Prometheus, OpenTelemetry, NVIDIA DCGM and OpenInference. It captures token counts, LLM‑as‑judge evaluations and span attributes for hallucination, guardrail violations and token overuse, correlates time‑to‑first‑token with GPU utilization and XID errors, and wires alerts to investigate anomalies. The stack produces actionable traces, logs and metrics for alerting and incremental AI‑guided remediation.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:01Who cares about LLM reliability
- 3:27Cloud-native AI observability challenges
- 4:51Failures in model training pipelines
- 6:15Hosting inference services
- 7:17Inference issues and hallucinations
- 9:26Observability foundation
- 11:55AI-native architecture overview
- 12:56Instrumentation and tracing
- 14:23Golden signals for AI systems
- 15:47Analyzing traces and metrics
- 16:50Evaluating LLM outputs
- 17:32Logs in AI observability
- 18:15GPU performance for inference
- 19:20Total cost of ownership
- 20:01AI troubleshooting trends
- 20:45Q&A
related talks
23mObservability for LLMs: Understanding What’s Happening Under the Hood
26mReducing Hallucinations and Evaluating LLMs for Production
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
42mStop Treating LLMs Like REST APIs
30mFrom Black Box To Insight: Observability for AI Agents in Production
32mEnd-to-End Observability for LLM Inference: From Token To GPU
33mLearn How to Run an LLM Inference Performance Benchmark on NVIDIA GPUs
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...