Democratizing Data Pipelines: Introducing the AI-Powered ETL Assistant
summary
A working ETL assistant that uses hundreds of existing pipeline examples plus a knowledge upload to an LLM to translate natural-language requests into cron expressions, SQL, task sequences and CREATE TABLE DDL, then invokes the catalog tool to create the table. UI uses Streamlit, metadata comes from OpenMetadata, and the LLM (via OpenAI assistant, LangChain and a vector DB) produces SQL which a QA agent validates against environment and schema rules. Teams report ~50–70% faster pipeline setup, with complex cases still requiring platform dev intervention.
AI-written summary · about
chapters (10)
- 0:00Introduction and problem
- 3:01Current ETL challenges
- 5:10AI assistant design
- 7:02Natural language scheduling
- 8:07SQL generation from questions
- 10:21Automated table creation
- 11:25Pipeline file generation
- 12:51Self-service pipeline execution
- 13:57Limitations and future work
- 14:40Key lessons learned
related talks
51mDeclarative ETL pipelines with Delta Live Tables
28mDynamic Data Pipelining with Luigi
16mEvolution of Extract-Transform-Load (ETL) processes towards data product pipelines
22mReliable data engineering made easy
42mBuild and automate a modern serverless data lake on
17mHow will AI transform your data platform?
27mAI-Powered ETL Pipeline Orchestration: Multi-Agent Systems in the Era of Generative AI
18mScalable, Code-Free ETL