How changes to Azure Synapse Analytics impact data architecture designs
summary
Many Synapse features now change architecture choices: use serverless first to avoid provisioning an Azure SQL DB and save about $1,200/month in simple scenarios; Spark is on 3.3 with smarter caching that reads/writes to the data lake about 30% faster; Delta Lake support adds time travel, MERGE and compaction for parquet files; R is available in preview but requires rewriting to use SparkR for distributed speed; Spark Advisor surfaces notebook performance issues and Azure SQL Link plus AAD improves integration.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker background
- 1:46Serverless databases vs traditional databases
- 3:49Shift from GUI to code in ETL processes
- 5:10Data Lake design and curated layers
- 6:13R language support in Synapse
- 8:04Improvements to Spark and Delta Lake
- 12:22Serverless improvements and Delta Lake tables
- 14:14Random feature additions and security
- 15:20Summary of key updates and future outlook
related talks
55mAzure SQL database, Azure Synapse Analytics, Azure Databricks?
54mOLAP on the Cloud with Azure Databricks and Azure Synapse
40mBest Practices and Optimization with the Spark Engine
52mConnect your data artifacts with Azure Synapse Analytics and Azure Purview
28mArchitecture patterns of Azure Cosmos DB & Azure Synapse Analytics
28mArchitecture patterns of Azure Cosmos DB
47mImplement a Serverless Data Analytics strategy with Microsoft Azure
42mAn Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric