RAG for Data Engineers: Why Your Metadata Is the Missing Piece
summary
Argues that retrieval augmented generation (RAG) closes the gap between generic LLMs and proprietary warehouses by indexing existing metadata (DDL, DBT model YAML, lineage, runbooks) into a vector store, demonstrating a month plan: week one audit, week two export your manifest JSON and index schemas, week three add a SQL agent, week four measure; prototype uses embeddings + Chroma DB and returns grounded SQL and incident fixes, built in ~15-20 engineering days with measurable faster incident resolution and fewer SME interruptions.
AI-written summary · about
chapters (13)
- 0:00RAG for Data Engineers
- 0:40Talk Roadmap
- 1:49Why LLMs Struggle
- 4:00RAG Under the Hood
- 6:07Data Team Advantages
- 9:33Four Workflow Use Cases
- 12:33Use Case Schema Q&A
- 14:45Use Case SQL Agents
- 17:47Use Case Incident Response
- 20:01Use Case Tribal Knowledge
- 22:34Metadata Is the Secret
- 24:09Month Long Action Plan
- 26:45Key Takeaways and Wrap
related talks
10mKeynote: Unlocking Corporate Data with Retrieval-Augmented Generation
44mRAG for a Medical Company: Technical and Product Challenges AI & Data
02h 47mFrom naive to advanced RAG: the complete guide
31mAdvanced Enterprise RAG Systems
42mTo RAG or not to RAG
53mFrom RAG to Riches: Secrets of Data-Augmented LLMs
42mDataEngBytes 2023 - SYD-T1-02 - Mikiko Bazeley
17mScaling Enterprise-Grade RAG: Lessons from Legal Frontier