Truth-seeker: Using LLM agents to build and verify knowledge bases
summary
Open-source LLM agents extract statements from documents, run web searches to score statements for sourcing and consistency, and tag each node as fact, opinion, or bias before adding it to a graph RAG knowledge base. Implementation uses smaller role-specialized models (mixl 8X 7B, llama 370b) and a faster inference service (Grock) to reduce cost. Results improve answer fidelity by corroborating claims against sources, but scraping limits, search costs, model bias, and consistency remain practical obstacles.
AI-written summary · about
chapters (18)
- 0:00Truth-seeker project intro
- 2:41Philosophy behind AI evaluation
- 5:11Knowledge and sources defined
- 7:01LLM tool-use frameworks evolution
- 9:12Open-source LLM model choices
- 11:21STORM paper inspiration
- 13:10Building and verifying knowledge bases
- 16:03Web data collection challenges
- 17:30Inference cost and availability
- 18:37Model consistency in opinions
- 19:48Future opportunities for agents
- 21:17Bias and user profiling risks
- 23:06Existing fact-checking limits
- 24:33Graph RAG next-step approach
- 25:14Ethical implications of models
- 27:53Q&A: Tool lifespan in business
- 29:04Q&A: Open-source vs commercial
- 31:13Q&A: Encouraging truth-seeking
related talks
23mKnowledge Graphs and LLMs in Action
32mMulti-Agent Query Generation w/ LLMs on Complex Knowledge Graphs
20mPractical GraphRAG: Making LLMs smarter with Knowledge Graphs
25mEnhance LLM Applications' Trustworthiness and Explainability
52mGive Your LLMs a Left Brain
28mLLMs Change — Where Should Knowledge Live? (Lessons From SBOM)
16mBeyond LLMs - The Next Frontier of Machine Reasoning
30mBuild an AI knowledge fabric for your organisation