Powering Interactive Data Analysis with Google BigQuery
summary
BigQuery runs ad hoc SQL over terabytes to petabytes with streaming inserts, table partitioning and schemas that accept nested JSON so unstructured event payloads become queryable. A Fluentd pipeline writes newline JSON to Cloud Storage, enables replay raw events from Cloud Storage for migrations, and loads to BigQuery where UDFs in JavaScript reshape rows. Storage costs about $20 per TB per month; a public-dataset run scans ~500 GB in ~10 seconds and costs ~$2.40. Examples include funnel analysis with window functions and Reddit/GitHub/StackOverflow public datasets.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 1:07Why big data matters now
- 4:47BigQuery as a solution
- 9:49Moving data to the cloud
- 13:50Using BigQuery with existing tools
- 18:15When to use BigQuery
- 21:50Serverless and scalability benefits
- 27:19JavaScript and custom reports
- 29:55Demo: JavaScript UDFs
- 31:05Public datasets and examples
- 33:57Conclusion and resources
related talks
35mScaling Interactive and Insightful Dashboards with Data Studio and BigQuery
28mFirebase analytics for BigQuery
47mAdvances in Stream Analytics: Apache Beam and Google Cloud Dataflow deep-dive
48mData Streaming with Apache Beam / Cloud Dataflow
38mLessons Learned from the Migration to Apache Airflow
41mModular & Reusable: A Recipe to Scaling Data Across a Portfolio
39mBuilding Modern Data Pipelines on GCP via a FREE online Bootcamp
41mReal-time Data Processing with Cloud Dataflow