Get insights from structured and unstructured data using the BigQuery DataFrames package
summary
BigQuery DataFrames demonstrates pushing pandaslike operations into BigQuery, implements over 700 pandas methods, and runs on scalable storage/compute; workshop processes ~3.5 million rows -> 154 samples (filters, grouping removed duplicates), generates text embeddings (all 154 succeed), clusters embeddings with K‑means 5 clusters, summarizes clusters with Gemini 1.5 Flash, and shows integration with BigQuery ML, Vertex AI, remote functions/Cloud Functions; code runs in notebooks and the pipeline completes end to end.
AI-written summary · about
chapters (10)
- 0:00BigQuery DataFrames intro
- 7:20Workshop environment setup
- 14:18Install and import package
- 21:47Reading structured data
- 31:35Pandas-style analysis
- 35:31Joining structured datasets
- 43:26Processing text embeddings
- 54:53K-means clustering
- 1:01:07LLM cluster summaries
- 1:06:33Cost and wrap-up
related talks
01h 29mPython + BigQuery + DataFrames: Hands on with scalable "serverless" analysis
40mDeveloping on Google Cloud with Python and DataFrames
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
41mAnalyzing Terabytes of Data with Google BigQuery
32mDataFrames: scaling up and out
27mNotorious B.I.G. Data: Wins and fails when building a big data platform in the cloud
25mSQL and TensorFlow: Enabling smart queries on a data warehouse
01h 17mPandas from the Inside / "Big Pandas"