Exploring open data with BigQuery
summary
Uses BigQuery to scan public sets like Wikipedia logs and Shakespeare, counting 39 million requests in one hour and scaling to 286 billion requests across a month. Queries that run regexes read columnar, compressed data spread across disks and complete heavy scans — one example reads 3.5 TB in 30s and returns 13,970 hits for “devoxx.” Execution uses a mixer/leaf node architecture (local regex/filtering on leaves, aggregation at mixers). Example recommendation query scans ~13 GB in ~7s, returns niche film suggestions; 13 TB of scans costs about $17.
AI-written summary · about
chapters (8)
- 0:00Start
- 5:11Open Source Implementations
- 6:12Managed Cloud Versions
- 12:13Qualities of a good RDBMS
- 14:25Storing data
- 20:38Follow the edits
- 21:19Pick a great movie
- 27:22Thank you!
related talks
41mAnalyzing Terabytes of Data with Google BigQuery
45mBuilding a Data Warehouse
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
59mGoogle BigQuery, Freebase, Maps:...
48mUsing Open Source Tech to Swap Out Components of Your Data Pipeline
40mFinOps Examples using Google Cloud BigQuery by Aliz.ai
24mHoney, I shrunk the bill!
24mForensic Log Analysis with BigQuery