AI Observability and Evaluation: Ensuring Trustworthy Intelligent Systems
summary
Concrete claims and artifacts not buzzwords: a custom Langfuse-based library with configurable rules, an on-screen demo dashboard that shows traces, per-conversation token and cost accounting, and PII/toxicity checks. OpenTelemetry and Azure integrations appear as sources, models compared include GPT4 and GPT5, and typical evaluation scores cluster around 0.4–0.7 evaluation scores not perfect ones. The system emits alerts, enforces governance, and measures tokens cost traces PII detection so teams can see what was built and whether it actually works.
AI-written summary · about
chapters (12)
- 0:00AI Observability challenges
- 1:25Custom evaluation frameworks
- 2:48Observability for AI systems
- 4:11Observability and evaluation tools
- 5:55Custom evaluation rules
- 7:41Security in AI agents
- 9:26Trustworthy AI pillars
- 11:11Demo: Custom evaluation library
- 13:17Interpreting metrics
- 15:24Human annotation frameworks
- 17:12Future of AI trust
- 18:16Q&A
related talks
41mFrom Lab To Life: Practical AI System Evaluation
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
40mOptimizing AI Applications for Production with Observability
01h 02mObserving AI Applications with OpenLit and OpenTelemetry
34mBuilding Trust with LLMs
38mAI Monitoring Mastery: Optimize Performance, Ensure Compliance, and Cut Costs
17mLLM Observability, Evaluation, Experimentation Platform
31mBuilding Trustworthy AI: Open Source Monitoring in Action