Building A Code Generated Data Platform
summary
Shows a runnable pipeline that uses Fivetran to land source data into Snowflake then auto-generates DDL from information_schema to create staging views, history tables and insert and merge tasks that implement CDC via row hashes and is_current flags. Demo loads 10 rows, then adds 5 and updates 2, producing 2 expired historical rows and new current rows; tasks run under a master batch and are scheduled hourly, proving the code-generation approach works end-to-end.
AI-written summary · about
chapters (7)
related talks
01h 20mSupercharge Your Real-Time Analytics: Snowpipe Streaming & Dynamic Tables in Snowflake
27mDataOps & Full Snowflake Automation In 1 Click
32mScalable Python and SQL Data Engineering without Migraines
31mData Mesh & Data Vault
25mPerfecting Our Modern Data Stack Recipe: What We Learned On The Journey At Huel
24mA Single Data Platform for All of Your Workloads
32mDataEngBytes 2023 - BRI-T1-10 Rishu Saxena & John Cosgrove
23mDelivering Retail Analytics & Data Projects With Data Created For Purpose