Building a Local News RAG: The Quest for Trustworthiness
summary
A small five-person team (data scientist, cloud engineer, product manager, journalist, UI/UX) builds a local-news RAG for a 1.5-year archive (~500k articles), finds a 100-article prototype breaks when scaling to thousands, and switches from OpenAI to Gemini embeddings. Key failures are hybrid search confusion on semantically identical weekly reports, catastrophic entity matching merges that falsely attribute crimes to local people, and tense/date errors; mitigations include query rewriting, geo-coding location hierarchies, time-slicing retrieval, prompt steering, shadow testing and ML-powered judges for faithfulness, hallucination, context relevance, and relevance.
AI-written summary · about
chapters (9)
- 0:00Introduction and team
- 3:40Challenges in news data
- 6:53Locality in RAG implementation
- 8:20Entity matching pitfalls
- 11:13Location metadata issues
- 14:28Handling dates and time
- 18:54Trustworthiness and evaluations
- 25:22Lessons learned and future work
- 29:17Q&A session
related talks
36mDon't tell me RAG is easy
40mNews Crawler via Langchain.RB and Gemini APIs
19mImproving RAG Systems - Key Challenges and Solutions
53mBuild RAG from Scratch
41mFrom RAGs to riches: Build an AI document interrogation app in 30 mins
43mHarnessing AI to strengthen trustworthy information
31mEvaluating RAG Applications: From Retrieval to Response Quality
27mFrom RAGs to riches: Build an AI document interrogation app in 30 mins