Data lakes beyond clusters
summary
Demonstrates running end-to-end analytics on Azure Data Lake without managing clusters: creates Data Lake Storage, uses U-SQL + cognitive extensions and the local emulator, uploads a 3–4 GB Cargill donations CSV (schema-on-read), compiles jobs into C++/.NET vertices, and runs sentiment and key-phrase extraction to Power BI. Uses adjustable pay-per-job analytical units (AU counts map to cores/GB), job graphs and profilers expose skew and cost, and findings show most funds concentrate in CA/NY and sentiment does not predict funding.
AI-written summary · about
chapters (31)
- 0:00Introduction
- 2:07Who is Dom
- 2:35Plan
- 3:46Data like
- 5:36Understand what youre after
- 6:49Supporting data sources
- 7:08Data lakes
- 7:48Storage and analytics
- 12:41Clusters
- 13:10Head nodes
- 13:30Worker nodes
- 15:42Complex infrastructure
- 16:12Cluster categories
- 17:09Doityourself cluster
- 17:49Cloud hosted
- 19:07Data Breaks
- 21:13Onprem
- 22:14Azure Data like analytics
- 25:28What is Scope
- 25:44What is Sequel
- 30:30Azure Data Like Analytics Resource
- 31:38Azure Data Like Analytics Dashboard
- 33:22Cognitive Extensions
- 37:01Demo of Cognitive Extensions
- 43:42Monitoring jobs
- 51:11Power BI
- 53:11C and native jobs
- 54:56From code
- 56:11Rookie mistake
- 56:41Recap
- 57:36Why should we bother
related talks
01h 02mAnalyzing StackExchange with Azure Data Lake
56mDataDevOps for the Modern Data Warehouse on Microsoft Azure
02h 05mBig Data and the Data Lake from the Analyst Perspective
25mBuild open format data lakehouse
11mBreaking Data Silos: Enterprise Data Management
26mDask-SQL: Empowering Pythonistas for Scalable End-to-End Data Engineering
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
14mCross-Cloud Campaign Analytics Meets JavaScript