Designing performant and scalable data lakes
summary
Shows how an enterprise data lake on Azure uses Azure Data Lake Storage plus Azure Databricks/HDInsight and Azure Synapse to ingest raw data, apply schema on demand, and serve BI or real‑time apps; recommends organizing by raw/enriched/curated/workspace zones, using parquet and coalescing small files to hit the 100 MB rule of thumb for throughput, and tuning row‑groups to cut IOs from tens of thousands to thousands; covers AAD + RBAC + POSIX ACLs, private endpoints, lifecycle tiers and choosing LRS ZRS GRS GZRS per environment.
AI-written summary · about
chapters (15)
- 0:00Start
- 2:28Traditional on-prem analytics pipeline
- 4:53Modern data warehouse
- 8:13Advanced Analytics
- 9:00Realtime Analytics
- 17:01Azure Data Lake Storage Summary
- 18:55How do I set up my data lake?
- 25:13How do I organize my data?
- 29:31How do I secure my data?
- 33:14A Little More on Authorization
- 39:50How do I manage cost? (Continued...)
- 42:32How do I optimize my data lake?
- 44:39File size and format
- 49:15Partition your data for optimized access
- 52:11Query Acceleration (Preview)
related talks
40mData Lake House in the Cloud: The Azure Solutions
51mHow to Build a Modern Data Lakehouse?
36mData Lakes with Azure Databricks
43mManaging a Data Lake on AWS
50mBest practices for implementing a modern data architecture with Azure
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
24mREDLake Azure - Enterprise Data Lake and Business Satellites
43mData Lake beyond the Buzzword