Analyzing StackExchange data with Azure Data Lake
summary
Processes a 150 GB+ StackExchange dump by uploading XML to Azure Data Lake Store & Analytics, deploying a custom C# XML extractor (NuGet DLL) to produce CSV aggregates, then matches user Location fields against ISO‑3166 friendly names (exact‑match only in the demo) and exports two CSVs consumed by Power BI map visualization. Jobs run serverless with tunable analytics units (cost ~£0.28/0.28 currency cents per AU‑minute), require scaling to avoid bottlenecks, and produce usable country and reputation summaries.
AI-written summary · about
chapters (15)
- 0:00Start
- 5:03Characteristics
- 7:53Security
- 9:11Azure Data Lake Store vs Blob Storage
- 10:59Azure Data Lake Analytics
- 12:11Data Sources
- 13:03Writing U-SQL scripts
- 14:25Extensibility
- 14:57Metadata Model
- 17:29Resource Management
- 18:13Pricing
- 20:09How is it setup?
- 53:29Azure Data Lake tools for Visual Studio (Code)
- 53:52Integration with Azure Services
- 58:10Summary
related talks
01h 02mAnalyzing StackExchange with Azure Data Lake
51mAzure Data Explorer - a Torch in the Dungeon of Observational Analytics
47mImplement a Serverless Data Analytics strategy with Microsoft Azure
03h 12mEMEA Live Stream
50mADX and the last crusade
42mThe Most Powerful Azure Service You’ve Never Heard Of
52mPower BI + Azure Synapse Analytics. All you need to start creating a unified analytics solution
01h 03mFeed Your Data! OData Data Feeds for PowerPivot Users