Evaluating multi-turn conversations
summary
Claims a concrete, three‑stage framework — experimentation, live monitoring, post‑production — with implemented Python pipelines, RAG precision/recall and hallucination checks, tool‑call accuracy tests, n+1 evaluation and simulation suites. It shows aligning LLM‑as‑a‑judge to human labels (calibration step) lets automated scores be trusted, warns that synthetic data overfits, and reports lessons from 110+ production projects. Live alerts (Slack), logging of prompts/tools/context, and regression gates are built and used to catch failures.
AI-written summary · about
chapters (12)
- 0:00Evaluating multi-turn conversations
- 2:32Evaluating LLMs and AI agents
- 5:39What to evaluate in AI agent conversations
- 7:25Evaluating RAG and tool calls
- 9:52Multi-turn evaluation methods
- 14:08LLM Evolve framework explained
- 15:54Experiment stage: evaluators and alignment
- 19:28Live monitoring and alerts
- 21:13Post-production review and KPIs
- 24:24Best practices and mistakes
- 30:44Key takeaways
- 31:47Q&A
related talks
26mReducing Hallucinations and Evaluating LLMs for Production
01h 26mShipping AI That Works: An Evaluation Framework for PMs
38mMaking LLMs reliable - A practical framework
46mEnhancing Agent Reliability Through Evaluation-Driven Development
22mValidating AI Outputs at Live Events
41mFrom Lab To Life: Practical AI System Evaluation
01h 28mMastering RAG Evaluation with MLflow & Open Source LLMs
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work