Make your LLM app a Domain Expert: How to Build an Expert System
summary
Claims SOTA clinical reasoning and production deployment for health insurers covering 50 million lives by building an adaptive system rather than swapping models. Uses a failure mode ontology, production failure‑mode datasets and quantifiable domain evals so clinicians label errors, suggest domain knowledge (there’s a button to add it), and PMs prioritize work. Engineers iterate against those real eval sets, pushing a baseline from ~95% up toward ~99% with same‑day fixes and tracked regressions.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:16The last mile problem
- 2:45Example
- 5:04The System
- 5:28The Models
- 6:32The Adaptive Domain Intelligence Engine
- 7:05Measuring Domain Specific Performance
- 9:14Domain Expert Dashboard
- 10:33Domain Expert Improvements
- 11:28Performance Visualization
- 12:00Domain Expert Suggestions
- 14:24Domain Expert Reviews
- 14:38How to Define a Domain Expert
- 16:34Putting Everything Together
- 18:06Final Takeaway
related talks
28mThe 10 Commandments for Crafting Custom LLMs with Domain Mastery
44mProduction-Grade LLMs & AI Agents
25mHow to Leverage Domain Expertise
33mDesigning LLM Applications for Massive Datasets
25mLLM Agents for Site Reliability
35mEvaluating LLM systems: the nuts and bolts
43mAI Development Lifecycle: Learnings of What Changed with LLMs
56mDehumanizing Agents: Why Explainability is Crucial in the LLM Era