Technical: Harvesting Trust: Methods for Validating Chatbots in
summary
A conversational CropWise AI agent ties foundation LLMs to three tool types—APIs, RAG retrieval, and a code-execution framework called repel—and an orchestration layer routes actions. A reproducible evaluation pipeline uses a custom controls database, repeated prompts to measure nondeterminism, and generates a trace file per run; component scores (API/RAG/repel and end-to-end) are published. Example: entity recall hits 50% while final selected-item accuracy can be 0% in a failed SQL-style query, guiding fixes.
AI-written summary · about
chapters (13)
- 0:00Validating chatbots in agriculture
- 3:08Quality assurance for AI trust
- 8:00Validating large language models
- 11:58AI in modern agriculture
- 14:27CropWise AI for farmers
- 15:11CropWise AI architecture
- 19:27Evaluation strategy for AI agents
- 21:54Complexity vs speed in AI
- 23:00Evaluation framework overview
- 26:56Scalable evaluation benefits
- 27:17Testing AI response consistency
- 29:28Text-to-SQL evaluation
- 32:43Q&A
related talks
47mTesting Conversational AI
42mQA for ML: How we can trust AI with Food Sustainability
46mTalking the Talk: Measuring Chatbot Accuracy
41mAutonomous Quality Engineering: AI Systems That Generate Execute Heal Learn and Govern Quality
36mChatbots Past, Present and Future
18mDefending the unknown using AI and cybersecurity
59mMaking the most of artificial and human intelligence for data science
46mCan you trust your AI?