Using DBT and Python to Build Production Pipelines in Snowflake
summary
A production ML pipeline converts notebooks and legacy scripts into runnable assets using DBT + Python in Snowflake, with models executed inside Snowflake and results materialized as incremental tables. The team wires CI/CD via GitHub CI/CD & Airflow, lets analysts contribute SQL heuristics, and reports faster delivery and lower ops overhead, while noting limits: custom Python packages and external logging are hard, SQL sprawl becomes brittle, and portability off Snowflake requires significant rewrites.
AI-written summary · about
chapters (7)
- 0:00Introduction and background
- 4:52From POC to production
- 8:29Choosing Snowflake and DBT
- 11:42Python integration in DBT
- 13:30Building the pipeline
- 16:01Pros and cons of the approach
- 18:36Tips for production pipelines
related talks
35mData Pipelines for Operational Metrics
49mdbt: A Software Engineering Approach To The Analytics Workflow
32mScalable Python and SQL Data Engineering without Migraines
21mMachine Learning in your Data Warehouse using Python
28mAutomating Database Operations with Cortex Code
29mScaling Python-Powered Machine Learning With Snowflake
23mAcceleratinag Data Science and Machine Learning with Snowflake DataTech
30mA Devops Approach to Data: A Tale Through the Trenches in Search of a Cloud Data Revolution