Implement a Serverless Data Analytics strategy
summary
Concrete outcome first: Azure storage Gen2 holds month of event files that are partitioned by type/year/month/day and 35 reports merged into single Parquet files using Synapse serverless copy jobs. Ingestion options compared: raw JSON landing, Parquet conversion, and append-block API for single large files. Transformation paths shown: UI data flows for aggregations, Spark pools and notebooks for Spark tables, and Delta for updates when mutability is required. Also notes parallel for‑each copies, schema drift failures, and cost/throughput tradeoffs with serverless Synapse SQL pool.
AI-written summary · about
chapters (24)
- 0:00Serverless data analytics intro
- 1:15Big data on Azure journey
- 3:03From relational to single source
- 4:32Dynamic JSON schemas in SQL
- 6:21Data movement and landing storage
- 8:26Data ingestion demo
- 10:14Mock API reports ingestion
- 13:57JSON to Parquet conversion
- 15:02Appending data with blob API
- 16:54Partitioned data in Azure Data Lake
- 17:59Synapse serverless SQL queries
- 20:51Data consolidation copy approach
- 22:20Azure Data Factory setup
- 24:09Merging files to Parquet
- 26:07Metadata discovery consolidation
- 30:13Parallel data copy faster
- 32:24Transforming data with Azure Data Flow
- 35:16Aggregating to SQL database
- 37:02Synapse Spark pools overview
- 38:49Spark tables creation
- 41:03Delta format for updates
- 42:35Delta Lake transactions
- 44:02Stable data strategies takeaways
- 45:58Q&A
related talks
47mImplement a Serverless Data Analytics strategy with Microsoft Azure
43mImplement a Serverless Data Platform with Microsoft Azure
30mProcess large-scale data and computational function in serverless manner
01h 01mServerless Azure Data handling
25mA simple serverless data pipeline
32mBuilding and scaling a log analytics platform - a serverless approach
42mBuild and automate a modern serverless data lake on
46mServerless Deployment of Analytics