Python + BigQuery + DataFrames: Hands on with scalable "serverless" analysis
summary
Hands-on use of BigQuery DataFrames shows a pandas-like API that keeps data in the cloud (no install, couponed GCP project) and runs on a real dataset of about 30 million rows (~7.5 GB). Examples include cleaning ZIP codes (50314 vs 5314), aggregating by ZIP and county, fitting a linear regression with BigQuery ML (R² ≈ 0.6), sending a local Series for prediction, and deploying a Python lemmatizer as remote functions via Cloud Functions (deployments take minutes and work).
AI-written summary · about
chapters (8)
- 0:00Python BigQuery DataFrames intro
- 8:03Demo environment setup
- 15:28Iowa alcohol sales exploration
- 34:51Data cleanup and ZIP analysis
- 45:56ML with BigQuery population
- 1:00:23Deploy text lemmatization function
- 1:15:39Liquor categories and county insights
- 1:27:07Summary and next steps
related talks
40mDeveloping on Google Cloud with Python and DataFrames
01h 08mGet insights from structured and unstructured data using the BigQuery DataFrames package
32mDataFrames: scaling up and out
50mServerless SQL Pipelines
32mScalable Python and SQL Data Engineering without Migraines
41mAnalyzing Terabytes of Data with Google BigQuery
35mData Processing Pipelines for Small Big Data
27mNotorious B.I.G. Data: Wins and fails when building a big data platform in the cloud