Everything You Need To Know About Agent Observability
summary
Raindrop claims production agents need explicit and implicit signals and shows concrete tools: tool error rate, latency and cost as explicit signals, plus user frustration classifier and regex for implicit signals. A/B experiments reduce frustration from 37% to 9% after a prompt change, and usefulness begins once a few hundred events. A demo builds a coding agent where write calls fail and a small self-diagnostic report tool surfaces the bypass behavior, and a triage agent backfills and automates investigations and exports to BigQuery or Statsig.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:14Introduction and the problem of agent failures
- 1:48Moving from evals to production monitoring
- 3:33The two types of signals: explicit and implicit
- 4:47Using classifier signals for observability
- 6:38Leveraging regex for signal detection
- 7:30Using experiments to validate improvements
- 9:42Q&A session: Statistical relevance and experimental design
- 16:07Introduction to self-diagnostics
- 20:15Workshop: Coding agent demonstration
- 24:01Live demo: Triggering and handling tool failure
- 30:26Best practices for self-diagnostic implementation
- 32:20Q&A: Real-world use cases and triage
- 40:02Q&A: Managing fast-paced experimentation
- 44:21Q&A: Trace visualization and data export
related talks
17mHow Observability-First Development Lets You Ship Agents in Weeks...
30mAgents Building Agents
29mLicense to Observe - Why observability solutions need agents
28mDO WE STILL NEED TO "OBSERVE" THE FUTURE OF AI & OBSERVABILITY
45mAI agents in practice - beyond the prototype
11mObservability for the AI Cloud
46mObservability in the real world: Improve systems w/iterative approach
16mAI Agents for Incident Investigation: The Good, The Bad, and The Ugly