Observe 2026
as the organisers put it
Arize Observe | June 4, 2026. The AI agent & evaluation conference showcasing Arize AX and Arize Phoenix, leading AI observability platforms
who it's for
AI agent engineers, machine learning engineers, ML platform engineers, prompt engineers, evaluation engineers, AI product managers, site reliability engineers
topics
trace-based observability and fixing; agent-as-judge evaluation; LLM-as-judge evaluation; Arize Phoenix runtime; prompt learning loop for agents; context management for long sessions; on-device model benchmarking (Llama/Gemma/Qwen); scalable LLM evaluation pipelines; golden dataset meta-evaluation; telemetry for control-flow and router correctness
| title | author | length |
|---|---|---|
| From Signal to PR: Anatomy of a Self-Improving Agent | Jason Lopatecki | 21m |
Arize shows that the actual unlock is traces on a filesystem pulled as files into the repo (sometimes ten megabytes), letting an agent produce a real fix —… | ||
| The Future of Evals: From LLM as a Judge to Agent as a Judge | Aparna Dhinakaran | 06m |
Arize reports running 100 million evals per month and sees teams run about 12 eval jobs on average with top teams at 3,800 evaluators, and argues evals must… | ||
| Frontier results, on device | Rachel Nabors | 31m |
Comparative evals use Phoenix and production traces from Mima to show local models replacing frontier calls: testing Gemma, Qwen, Llama 3.2 against Claude… | ||
| LLM Observability, Evaluation, Experimentation Platform | Dat Ngo | 17m |
Shows concrete claims and tools: telemetry catches an agent calling B before A where B depends on A, five flavors of eval signal (LLM judge, human feedback… | ||
| Ship Real Agents: Hands-On Evals for Agentic Applications | Laurie Voss | 02h 04m |
Builds a financial‑analysis agent, instruments it with trace with Phoenix, and shows exact checks: simple code evals, built‑in and LLM‑as‑a‑judge rubrics, and… | ||
| How we solved Context Management in Agents | Sally-Ann Delucia | 16m |
The team finds that simple truncation and naive summarization fail: over-truncation breaks reasoning and summarization gives the LLM too much control. The… | ||
| Build a Prompt Learning Loop | Sally-Ann Delucia, Fuad Ali | 52m |
A hands‑on loop that generates outputs, runs OpenAI + Elements judge evaluations, collects rule‑level failures and English explanations, and injects those into… | ||
| Shipping AI That Works: An Evaluation Framework for PMs | Aman Khan | 01h 26m |
Concrete claim: a reproducible eval loop (dataset → prompt playground → experiment → LLM-as-judge → human labels) surfaces real gaps. Demo builds a multi-agent… | ||
| The Unreasonable Effectiveness of Prompt Learning | Aparna Dhinakaran | 11m |
Claims a feedbacK-driven system-prompt learning loop that applies RL-style updates to prompts not weights and auto-tunes an agents.md. Uses unit tests +… | ||
| Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work | Dat Ngo, Aman Khan | 25m |
Practical recipes: build evals that grade actual claims, not topics — combine LLM-as-a-judge with human-in-the-loop and cheap heuristics; use encoder-only… | ||
| Ensure AI Agents Work: Evaluation Frameworks for Scaling Success | Aparna Dhinkaran | 15m |
Claims a concrete evaluation stack: routers, skills, memory instrumented as traces; metrics include router correctness rate, whether skill calls pass the right… | ||
| nothing matches | ||