Tutorial: Building an Observable AI Agent With an Open Source Stack
Talkat Open Source Summit Japan + AI_dev: Open Source GenAI & ML Summit + Automotive Linux Summit 2025 · 2025-12-08..12-10·
summary
Builds a working agent (a React agent orchestrated with LangChain) inside the OpenTelemetry demo shop, instrumented with OpenTelemetry traces and manual spans (product ID) and with Langfuse v2 capturing LLM-level prompts, responses, costs and per-call latencies. Uses an OpenAI model plus a judge LLM that returns JSON scores 0–1 with comments, runs in Docker, and the end-to-end demo produces connected traces, cost breakdowns, and evaluation metadata — the example runs and stores traces locally for hands‑on follow‑along.
AI-written summary · about
chapters (8)
- 0:00AI agents and observability
- 4:23AI agent and ReAct model
- 9:00Building AI agent demo
- 12:14LangChain tools and agent
- 14:44OpenTelemetry instrumentation
- 20:15Langfuse for LLM observability
- 24:47Evaluating responses with LLM-judge
- 26:59Summary and hands-on
related talks
31mDeep Dive To AI Agent Observability
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
01h 02mObserving AI Applications with OpenLit and OpenTelemetry
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
18mWhat did AI just do? Making LLMs Observable with OpenTelemetry
53mHands-on with AI agents: From scratch to LangGraph
02h 08mIntroduction to AI Agents