How to pick the best Data transformation strategy
summary
Recommends choosing tools by concrete tradeoffs: Power BI works for fast delivery but loses source traceability, forces Premium for dev pipelines (~$5k/month) and limits refresh cadence; ADF/Synapse give CI/CD but visual data flows are opaque, costly and can time out on huge files (lab hit a 25‑minute timeout on a 3 TB import); PySpark/Databricks handles complex parsing and parquet well when code is modular and documented. Practical pattern: land raw, transform to structured, curate dimensional tables, use serverless SQL for ad hoc (~$5 per TB scanned) and enforce CI/CD and standards.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker intro
- 5:20Key goals for data transformation
- 10:15Reality of data transformation projects
- 13:44Power BI data modeling pros and cons
- 21:26Azure Synapse and ADF strategies
- 26:41Pi Spark for ETL and transformation
- 32:39Design patterns for reliability
- 42:05Summary and best practices
- 43:08Q&A and closing remarks
related talks
01h 01mCurrent Methods for Building Data Pipelines
56mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
35mData Processing Pipelines for Small Big Data
19mDemystifying Modern Data Pipeline Architecture
30mData Engineering building your BI infrastructure from scratch
58mTales of Data Architecture Evolution
41mModern Data Stack: Optimising and Scaling Data in a Tech Company
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics