A low cost serveless synapse data platform
summary
A minimal data platform built on serverless SQL and a Parquet data lake uses an optional Azure SQL metadata table plus Synapse pipelines to generate layered bronze/silver/gold results; transformations use CREATE EXTERNAL TABLE AS SELECT and are metadata-driven pipelines rather than Spark or dedicated pools. It cannot write Delta, struggles with nested JSON and streaming, and introduces debugging and governance overhead, but processes small-to-moderate datasets cheaply and suits refresh rates roughly once per day or less.
AI-written summary · about
chapters (15)
- 0:00Introduction to low-cost Synapse
- 1:24Why this concept exists
- 2:47Building blocks of the platform
- 3:50Synapse components used
- 5:14Data platform layers explained
- 6:18Processing options overview
- 8:04Metadata-driven pipelines
- 10:12Parameterized pipeline example
- 12:42Real-world implementation tips
- 14:06Cost drivers and savings
- 15:51Key challenges to consider
- 17:17Limitations of serverless SQL
- 18:42Maintenance and governance
- 19:25When to use this approach
- 20:09Benefits and next steps
related talks
58mAzure Data Lake & Synapse Server less Architecture
30mProcess large-scale data and computational function in serverless manner
49mAzure Synapse Analytics
53mAzure Synapse Analytics: Evolving Azure SQL Data Warehouse
55mAzure SQL database, Azure Synapse Analytics, Azure Databricks?
19mNever ETL again, thanks to Synapse Link. Really?
24mSynapse Analytics
54mAn introduction to Azure Synapse Analytics!