LLMs gone wild
summary
Shows why retrieval augmented generation (RAG) often fails in production: embedding mismatch, bad chunking, messy PDFs and images, stale vector indexes, and adversarial queries. Demonstrates real tools and mitigations — text-embedding-ada-002, LangChain, Chainlit, LlamaParse, Azure Document Intelligence, vector DBs, chunkwiz — and uses numbers: LLM generation ~60–85% accuracy, retrieval ~up to 85%, yielding about 72% end-to-end correctness. Recommends parsers, metadata filters, hybrid search, reranking, query expansion and agent tools instead of blind augmentation.
AI-written summary · about
chapters (14)
- 0:00Introduction to LLM pitfalls
- 2:00LLMs as probabilistic systems
- 4:42Understanding RAG fundamentals
- 6:32Embeddings and semantic search
- 10:55Building a RAG application
- 18:20Demo: RAG in practice
- 21:53Production risks and challenges
- 27:06Evaluation frameworks
- 31:21Optimizing data and chunking
- 36:49Advanced retrieval techniques
- 41:26Enhancing search performance
- 47:52Generation and AI agents
- 51:48Enterprise RAG reality check
- 55:42Conclusion and safety
related talks
01h 00mLLMs gone wild
36mTechniques to avoid common LLM pitfalls
51mAvoid common LLM pitfalls
18mLLMs and GenAI: A Guide to Building Your Own RAG Model for Document Interaction
48mGoing Beyond Prompt Engineering
28mNavigating Challenges and Enhancing Performance of LLM based Applications
53mFrom RAG to Riches: Secrets of Data-Augmented LLMs
01h 33mRAG How To Talk To Your Data