Prompt Failures and Latency Spikes: Observability for AI
summary
Demonstrates deploying OpenLit with Langfuse and a proxyed light LLM to capture trace-based prompt chains, surface per-call token counts and cost, and enforce virtual keys that limit models, budgets and rate limits for teams. Shows P95/P99 latency spikes mapped to spans, automatic hallucination checks using an LLM-as-judge, and dashboards that attribute spending and failures to sessions, prompts and models so teams can decide whether latency, variability or token cost caused a failure.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:48AI observability importance
- 8:27AI adoption challenges
- 14:11AI vs traditional observability
- 19:11Observability layers in AI
- 23:06Enterprise AI failure modes
- 33:10OpenLIT demo
- 37:50Light LLM virtual keys
- 46:03Langfuse for prompts
- 56:17Tool recommendations
- 59:11Q&A
related talks
29mYour Prompt Is Not a Unit Test — instrumenting LLM apps in production
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
01h 02mObserving AI Applications with OpenLit and OpenTelemetry
38mAI Monitoring Mastery: Optimize Performance, Ensure Compliance, and Cut Costs
29mLangfuse, OpenLIT, and Phoenix: Observability for the GenAI Era
05mLightning Talk: Your Prompt Is a Cross-Border Data Transfer
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
24mHow OpenTelemetry Helps Generative AI