AI Development Lifecycle: Learnings of What Changed with LLMs
summary
Claim: proof‑of‑concepts with LLMs are trivial but evaluation becomes the bottleneck; neglecting evaluation blocks production. Built a RAG pipeline for hotel operations that moves from ~30% to 88% performance through data prep, beta testing and iterative prompting. Used DVC, Google Sheets, chainlit and monitoring (Langfuse / Literal AI style tooling), preference for manual inspection over LLM-as-judge, and reduce variance by inferring five times per example to distinguish real gains from model noise.
AI-written summary · about
chapters (14)
- 0:00AI lifecycle changes with LLMs
- 1:34Why AI products fail in production
- 3:03NLP lifecycle overview
- 4:55How LLMs changed AI development
- 6:24Evaluation and monitoring challenges
- 8:33Hotel group case study
- 11:48Hotel group data preparation
- 13:56Clarifying product scope
- 16:07LLM evaluation methods
- 19:45Manual vs LLM-as-judge
- 21:53Performance over five weeks
- 24:22Production monitoring
- 28:59LLM lifecycle differences
- 31:54Q&A on citations
related talks
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
35mEvaluating LLM systems: the nuts and bolts
44mProduction-Grade LLMs & AI Agents
26mBecoming AI engineers
32mBeyond LLMs: Building an AI-Powered Decision System at Scale
27mAI Demystified Unveiling the Intelligence Behind the Machine
36mAdvanced Evaluation Techniques for AI Agents