Increasing the LLM Accuracy for Question Answering on Structured Data: Knowledge Graphs to the Rescu
summary
Benchmark uses an enterprise insurance SQL schema, a set of enterprise questions, an ontology and mappings that virtualize the SQL as an RDF knowledge graph and run SPARQL. Core findings: GPT-4 zero-shot on SQL: 16% execution accuracy, Knowledge Graph representation: 54% (≈3×), and an Ontology-based Query Check that detects domain/range and path errors plus an LLM repair loop raises overall accuracy to 72% (with 8% unknowns and a 20% error rate). Results come from reproducible prompts, virtualized KG rewrites to SQL, and execution-based scoring.
AI-written summary · about
chapters (23)
- 0:00LLM accuracy challenges
- 1:27Why knowledge graphs matter
- 3:13What is a knowledge graph
- 5:17Text-to-SQL vs knowledge graphs
- 7:01Benchmarking LLM accuracy
- 8:46Benchmark framework design
- 10:11Question and schema complexity
- 11:34Context layer role
- 13:19Experimental setup and prompts
- 15:02Measuring execution accuracy
- 17:09Results: 3x higher accuracy
- 19:57Validation of findings
- 20:59Why queries fail without graphs
- 22:01Graph query examples
- 24:11Semantic checks for accuracy
- 28:04Ontology-based query checker
- 31:20LLM repair and error reduction
- 33:064x accuracy with semantics
- 34:53Knowledge graph-powered system
- 36:5993% accuracy achieved
- 37:41LLM QA challenges
- 39:26Defining AI trust
- 40:28Invest in knowledge graphs
related talks
30mMaking Your Knowledge Graph LLM-Ready: Quality Assessment and Design Strategies for GraphRAG
41mEnhancing LLMs with Graph Technology
23mKnowledge Graphs and LLMs in Action
44mProduction-Grade LLMs & AI Agents
30mBuilding a HybridRAG Document Question-Answering System
21mAgentic GraphRAG: Autonomous Knowledge Graph Construction and Adaptive Retrieval
32mMulti-Agent Query Generation w/ LLMs on Complex Knowledge Graphs
27mLies, Damn Lies, and AIs