The 10 Commandments for Crafting Custom LLMs with Domain Mastery
summary
Focuses on turning proprietary data and domain expertise into deployed LLMs using an internal genos platform and a cross‑product assistant called in assist. Shows a concrete QuickBooks use case: text‑to‑invoice automation that speeds cash collection by ~100x and explains when to prefer prompt‑engineering, RAG, or costly fine‑tuning. Describes an internal leaderboard of domain tests with hundreds of questions, using expert-labeled datasets, LLMs-as-data‑generators, strict security/verification gates, continuous feedback loops, and mandatory human‑in‑the‑loop escalation.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker background
- 2:14Intuit and LM Excellence team
- 7:56Defining success criteria for LLMs
- 10:25Evaluation and measurement of models
- 13:20Starting with prompt engineering before fine-tuning
- 16:41Cross-domain expertise in LLMs
- 18:46Standardizing safety and security checks
- 20:17Continuous improvement and feedback loops
- 23:11Keeping humans in the loop
- 24:15Strategic investment in LLMs
- 26:23Adaptability and future-proofing
related talks
29mCustomizing and Evaluating LLMs, an Ops Perspective
15mLLMs for Data-Driven Decision Making in Product Development
27mThe Lean LLM development playbook
46mBuilding an LLM Infrastructure from Scratch: Lessons from X5 Tech
50mPimp My Llama - How to Customize LLMs for Your Business Case
33mDesigning LLM Applications for Massive Datasets
35mEvaluating LLM systems: the nuts and bolts
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications