Testing, Evaluating and Debugging Generative AI and Agentic Ap...
summary
Covers concrete tooling and practices for testing and debugging agentic GenAI: Langfuse end-to-end tracing for prompts, tool calls, latency and cost; OpenTelemetry for semantic conventions and full-stack correlation; MLflow for lifecycle tracing; DeepEval for 30+ automated metrics and synthetic test data plugged into CI/CD. Describes building incident recovery agents that use vector DB search, RAG and a dependency knowledge graph to propose probabilistic root causes and occasional automated remediation. Finds observability costs require truncation, masking and tail sampling to be practical.
AI-written summary · about
chapters (14)
- 0:00AI observability challenges
- 3:54AI vs traditional observability
- 6:23AI observability pillars
- 8:27Evaluation metrics for AI
- 10:12AI observability tools convergence
- 12:45MLflow AI observability
- 13:26OpenTelemetry in AI observability
- 15:11Langfuse tracing demo
- 19:32AI testing differences
- 20:59Generative AI testing
- 22:25DeepEval for LLMs
- 23:35AI monitoring conventions
- 25:20AI incident recovery
- 27:48Cost-effective AI observability
related talks
16mBuilding Applications with AI Agents
29mImpact of #GenerativeAI in Development and #SoftwareTesting
41mAI-Augmented Testing: How Generative AI, Prompt Eng Turn Testers Into Superheroes
41mGenerative AI: bliss or miss?
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
40mDavid Fogl: Leveraging Generative AI for Enhanced Software Testing
09mTesting Strategies for Generative AI Applications
35mThe Agentic AI Engineer