Analyzing StackExchange with Azure Data Lake

on YouTube ↗01h 02m

" summary

Uploads a 150+ GB StackExchange dump to Azure Data Lake Store, runs U-SQL jobs in Data Lake Analytics with a custom XML extractor to avoid shredding XML, converts many per-site XML files into a single aggregated CSV, and groups users by country to compute counts and average reputation. Uses PowerShell for ingestion, provisions Analytics Units (75 AU used then optimized to ~8), and builds a Power BI dashboard that loads the CSV. Outcomes: job succeeds, CSV produced, visualization works; storage and compute costs shown. 150+ GB dataset converted XML to CSV

AI-written summary · about

" chapters (27)

1. 0:00Introduction
2. 0:44Tom Kerkhove
3. 1:11Big Data
4. 1:56Big Data Analysis
5. 4:12Data Like Store
6. 6:31What is a Data Lake
7. 8:00Data Lake vs Warehouse
8. 8:47Security
9. 9:53Pricing
10. 10:19Blob Storage
11. 14:51DataLikeStore
12. 23:03XML to CSV
13. 23:17Azure Data Analytics
14. 24:40Data Stores
15. 25:45Script Overview
16. 26:51Code Behind
17. 27:11Metadata Model
18. 27:37Scripts
19. 29:16Security model
20. 29:37Application limitations
21. 30:22Visual Studio Code
22. 34:13Extract Script
23. 38:00Submit Script
24. 43:36Output Data
25. 46:03Cleaning Users
26. 49:36Job Overview
27. 51:21Optimized Script

" related talks

• 59mAnalyzing StackExchange data with Azure Data LakeTom Kerkhove · NDC Sydney 2017
• 01h 02mData lakes beyond clustersDom Raniszewski · NDC Sydney 2018
• 58mBig Data Analytics with Excel 2013Peter Myers · PASS Business Analytics Conference 2013
• 55mExploring StackOverflow dataEvelina Gabasova · NDC London 2017
• 51mAzure Data Explorer - a Torch in the Dungeon of Observational AnalyticsPawel Potasinski, Guy Reginiano · SQLBits 2023
• 47mImplement a Serverless Data Analytics strategy with Microsoft AzureRoberto Freato · DevDays Europe 2022
• 50mADX and the last crusadeFrank Geisler · SQLBits 2023
• 42mThe Most Powerful Azure Service You’ve Never Heard OfBradley Ball, Patrick LeBlanc · SQLBits 2023