Developing on Google Cloud with Python and DataFrames

on YouTube ↗40m

" summary

BigQuery is presented as a serverless analytics engine with a 250 petabyte single-query scale, result caching (~24 hours), and SQL plus Python execution paths: use Ibis lazy evaluator to compile DataFrame-style code to SQL (16 backends), test locally, or run pandas-like workflows via BigFrame (pandas-compatible API). Execution options include PySpark stored procedures (serverless Spark) and remote functions (Cloud Functions / Cloud Run) for per-row enrichment; demos show Iceberg/BigLake object tables and vision API calls that detect faces and PII.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 2:08Google Cloud open source
3. 4:14BigQuery overview
4. 10:35Data frames with Ibis
5. 14:12PySpark in BigQuery
6. 15:58Big Lake open formats
7. 18:08Python remote procedures
8. 20:57Unstructured metadata
9. 21:18Demo: Ibis queries
10. 26:10BigFrames API
11. 28:01Demo: PySpark execution
12. 31:10Demo: Object tables
13. 34:58Summary
14. 35:53Q&A

" related talks

• 01h 29mPython + BigQuery + DataFrames: Hands on with scalable "serverless" analysisHsia, Swena, Williams · PyData Global 2024
• 01h 08mGet insights from structured and unstructured data using the BigQuery DataFrames packageTim Swena · PyData NYC 2024
• 37mBuilding a Data Platform from scratchRodel van Rooijen · PyData Amsterdam 2024
• 45mBuilding a Data WarehouseArtem Nikulchenko · Software Architecture fwdays'22
• 33mManaging IoT Storage with Google's Cloud PlatformDaniel Bergqvist, Billy Jacobson · Google I/O 2019
• 31mGoogle Cloud Bigtable: petabyte-scale NoSQL database for scalable computationDmytro Timofeev · GDG DevFest Ukraine 2017
• 36mMigrating Our Data Warehouse to the CloudGerard Toonstra · Big Data Conference Europe 2019
• 33mBigQuery DeepDive: dissecting Google's Data WarehouseClément Bosc · Sunny Tech 2022