LLM Observability, Evaluation, Experimentation Platform
summary
Telemetry catches a real failure mode where an agent calls tool B before tool A because B depends on A, and traces reveal the execution path that code audits miss. The system defines five eval signals (LLM judge, human feedback, golden datasets, deterministic checks, business metrics) and scopes evals from span through trajectory and session. Arize Phoenix is open source and runs in a single container no Kubernetes required; the enterprise layer Alex scans traces, surfaces high latency and errors, and auto-creates evals to automate the observability loop.
AI-written summary · about
chapters (8)
- 0:00LLM observability intro
- 1:53AI observability fundamentals
- 4:41Agent performance & user satisfaction
- 7:08Five evaluation signals
- 9:14Tech & domain expert collaboration
- 12:27Experimentation workflows
- 13:28AI automation future
- 15:36Arize products & solutions
related talks
17mLLM Observability, Evaluation, Experimentation Platform
25mLLM Agents for Site Reliability
15mKeynote: Into the Black Box: Observability in the Age of LLMs
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge
19mExamining the Principles of Observability and Its Relevance in LLM...
38mMaking LLMs reliable - A practical framework
21mAI-Powered O11y with LLMs: Resilience in Cloud PaaS Archs
23mObservability for LLMs: Understanding What’s Happening Under the Hood