Taming Testing of AI apps
summary
Describes four concrete testing layers: fast CI-style checks that turn temperature down and demand structured output (JSON schema via LangChain4j), mid‑level evaluations using synthetic datasets and an LLM judge or re-ranker (CoHere, Mistral/Mistral‑class) with a pass threshold around 0.80–0.85, and production A/B with Langfuse monitoring. Demonstrates vector similarity tooling and recommends vector embeddings as the default comparator; live demo fills PetClinic form via a model and the end‑to‑end test reports pass.
AI-written summary · about
chapters (5)
- 0:00Testing AI apps intro
- 10:36Four AI testing techniques
- 24:53Level two AI evaluation tests
- 32:44Level three A/B tests
- 39:54Q&A
related talks
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
24mHow to Test AI Features Before Your Users Do
42mRobust Testing Strategies for AI-Driven Applications
55mAI-First Testing
25mAI vs. AI: How to Test AI Systems Using AI
16mTransforming Test Automation with AI and Machine Learning
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
22mFrom the Trenches: Automated Testing of Large Language Models