Data Quality Revolution : The Fabric-OpenAI Approach
summary
Demonstrates a working pipeline that pairs Fabric + Azure OpenAI to clean messy insurance data: notebooks call deployed models (DQ demo 1 = GPT‑4, DQ demo 2 = embeddings, DQ demo 3 = GPT‑3.5) to normalize dates, countries, addresses and flag missing fields; GPT‑3.5 date normalizer handles most formats quickly while GPT‑4 is slower (~18s for 10 rows) and costlier. Pattern reduces scope (often reduced from billions to millions) before LLM, validates outputs and sends failures to human review; code is in a notebook on GitHub.
AI-written summary · about
chapters (11)
- 0:00Introduction to the session
- 1:34Evolution of AI models
- 4:03Exploring Azure AI Studio
- 8:41Prompt engineering for data
- 12:31Integrating LLM into notebooks
- 18:40Data processing patterns
- 24:07Azure OpenAI service overview
- 29:00Microsoft Fabric fundamentals
- 33:29Fabric and AI demonstrations
- 40:52Managing AI output and security
- 45:22Advanced training and Q&A
related talks
27mDataEngBytes 2023 - PER-T1-08 - Sergio Zenatti Filho & Elizabeth Antoinne
22mModels to outcomes with end-to-end data science workflows in Fabric
46mIntegrating Azure AI and Fabric for Next-Gen AI Solutions
15mMapping the Future of Data & AI
01h 03mIt's All About AI
56mData, AI, Hype and Fear - Recipe for Disaster?
05mData Quality and Validation in ML Pipelines
46mFrom Data to Insights: Building AI-Powered Data APIs