Production-Grade LLMs & AI Agents
summary
An LLM system goes from 62% to 95%+ accuracy by layering fixes: prompt tweaks (62→68%), basic RAG (68→71%), fine-tuning (71→74%), semantic chunking (74→82%), then a Neo4j knowledge graph and an LLM evaluation loop to reach ~90–95% and eliminate hallucinations. Concrete failures include a refund answer saying 60 vs 14 days, a silent drop to 44% after dependency updates, RAG returning “upgrade” instead of “cancel,” costly retry loops, CI/CD evaluation gates, shadow traffic, and observability + access-control requirements.
AI-written summary · about
chapters (15)
- 0:00Production-grade LLMs & agents
- 1:26LLM production failures
- 4:59Improving LLM accuracy iteratively
- 7:51Semantic chunking & knowledge graphs
- 11:08Evaluation-driven loops
- 14:01Model accuracy & semantic failures
- 17:09Generative AI architecture
- 21:00LLM evaluation matrices
- 22:24LLM security risks (OWASP)
- 27:25Testing & security frameworks
- 29:10CI/CD for LLM evaluation
- 33:13Observability & feedback loops
- 37:28Agentic AI maturity model
- 40:38Private domain LLMs future
- 43:27Q&A
related talks
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
35mEvaluating LLM systems: the nuts and bolts
43mAI Development Lifecycle: Learnings of What Changed with LLMs
40mKeynote
36mLLMs and AI agents demystified
25mFrom Chaos to Control: Building Enterprise-Grade LLM Agents That Actually Work
32mBeyond LLMs: Building an AI-Powered Decision System at Scale
29mCustomizing and Evaluating LLMs, an Ops Perspective