LLM Observability, Evaluation, Experimentation Platform
summary
Shows concrete claims and tools: telemetry catches an agent calling B before A where B depends on A, five flavors of eval signal (LLM judge, human feedback, golden datasets, deterministic checks, business metrics), eval scopes from single span to session and trajectory, and an open source runtime Arize Phoenix is open source runs as a single container plus an enterprise layer where Alex scans traces and creates evals automatically aiming to automate the observability → evaluation → experimentation loop.
AI-written summary · about
chapters (8)
- 0:00LLM observability intro
- 1:53AI observability fundamentals
- 4:41Agent performance & user satisfaction
- 7:08Five evaluation signals
- 9:14Tech & domain expert collaboration
- 12:27Experimentation workflows
- 13:28AI automation future
- 15:36Arize products & solutions
related talks
17mLLM Observability, Evaluation, Experimentation Platform
38mMaking LLMs reliable - A practical framework
15mKeynote: Into the Black Box: Observability in the Age of LLMs
25mLLM Agents for Site Reliability
35mEvaluating LLM systems: the nuts and bolts
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
36mAdvanced Evaluation Techniques for AI Agents
25mEvaluating An LLM's Ability To Code