Lessons Learned From Implementing Generative AI Solutions
summary
Practical checklist of what actually breaks when shipping LLM features: language ambiguity causes silent errors and nondeterministic grades; hallucinations appear (a fake parliamentary-question generator once produced a terrorism-support claim; a code suggestion referenced a non‑existent pydicom attribute); inference costs and latency spike (examples: $53 for a batch of summarizations in one scenario, 25–40s average API latency in June); evaluations require human gold standards or LLM-assisted scoring; security needs moderation and NeMo Guardrails against prompt injection; production RAG requires vector DBs with embeddings, careful chunking, and LoRA/quantized PEFT experiments.
AI-written summary · about
chapters (12)
- 0:00Introduction and agenda
- 1:15Building blocks of LLM apps
- 3:00Language model basics
- 15:07Core challenges overview
- 15:29Language ambiguity issues
- 21:12Hallucinations in LLMs
- 25:17Cost and inference challenges
- 29:13Latency and performance
- 33:09Evaluation difficulties
- 38:10Security and bias risks
- 43:06Retrieval augmented generation
- 51:51Future outlook and summary
related talks
57mLLMs gone wild
13mNew Challenges in Software Quality Brought by GenAI and LLMs
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
01h 49mAI engineering with open access LLMs that lie, curse, and steal
32mBeyond LLMs: Building an AI-Powered Decision System at Scale
46mEnhancing Agent Reliability Through Evaluation-Driven Development
40mMeasuring the unmeasurable: How to evaluate generative AI-based systems?
15mChallenges in moving Gen AI apps to production