Data Lakes with Azure Databricks
summary
Shows how to build a data lake on Azure using Azure Databricks + ADLS Gen2 and Event Hubs for Kafka with concrete code: an external table over CSV, convert to delta format partitioned by year/month, then run queries. Counts take ~16.5s on raw CSV, top-10 queries on delta return <1s, and a batch ETL processes ~80 million rows in ~5 minutes. Demo also runs a Kafka-style streaming job that writes many small files and recommends delta optimize and compaction.
AI-written summary · about
chapters (11)
- 0:00Introduction to data lakes
- 9:51Demo querying data lake
- 16:49Reference architecture overview
- 19:59What is Apache Spark
- 22:46Azure Data Lake Storage Gen2
- 23:47Event Hubs for streaming
- 26:32Loading data batch example
- 28:38Loading data streaming example
- 30:43Storage best practices
- 32:51Delta Lake benefits
- 34:36Final thoughts and resources
related talks
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
40mData Lake House in the Cloud: The Azure Solutions
51mHow to Build a Modern Data Lakehouse?
55mDesigning performant and scalable data lakes
58mBuild reliable data lakes with Delta Lake & Databricks
20mReady. Set. Automate! Have your Azure Databricks Lakehouse Up and Running in 20 Minutes!
21mBuilding Data Integration Services In Real Time
45mSimplify your ETL with Databricks Delta Lake