Practical tactics to build reliable AI apps
summary
Shows a concrete workflow: start POC that works ~50% of the time, build reverse-engineer real-world evals before iterating, use o3/LLMs to generate 50+ paraphrases and a checklist per question, and run frequent experiments to see which change (model, prompt, logic, or data) fixes a failing eval without causing regressions. Emphasizes specific metrics like measure escalation rate for support bots and using a mock database for text-to-SQL to get deterministic, actionable benchmarks.
AI-written summary · about
chapters (7)
- 0:00Introduction to reliable AI
- 1:29Why AI apps are unreliable
- 3:15Wrong metrics for AI solutions
- 4:43Real-world scenario evaluations
- 8:39Building evaluations early
- 11:52Tailoring evaluations to solutions
- 13:37Key takeaways and summary
related talks
46mEnhancing Agent Reliability Through Evaluation-Driven Development
27mBuilding AI applications: Successes and Lessons
41mSurviving and Thriving with AI in QA
24mHow to Test AI Features Before Your Users Do
27mHow to Build AI Agents and Keep Your Sanity
01h 26mShipping AI That Works: An Evaluation Framework for PMs
14m15 Months of building AI agents in 15 minutes
38mAI Just Happened to Me : Help!!