Honey, I shrunk the bill!
summary
Examples land fast: a single 22‑second query costs $10K, a two‑week data‑lake job runs $75K, a forgotten VM bills $45K, and one BigQuery run hits $4K on an 800TB table. Audit, monitoring and fixes actually reduce spend — partition and cluster tables, use job views and BigQuery execution graphs, set up Prometheus/Grafana/ELK, Soda or Monte Carlo for profiling, implement checkpoints and delta tracking, automate shutdowns and GPU monitors, and use Triton with server‑side batching — proven to save hundreds of thousands.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:26Speaker and agenda
- 2:29Minimum order framework
- 3:32Pre-audit setup
- 4:35Pre-audit governance
- 7:04Audit practices
- 9:11Post-audit optimization
- 10:58ML pipeline costs
- 12:23Data layer traps
- 16:11Compute inefficiencies
- 20:48GPU underutilization
- 22:33API logs and egress
- 23:36Closing and key takeaways
related talks
20mAtomics Habits in cost optimizations in the cloud
50mServerless SQL Pipelines
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
22mUsing BigQuery as a data warehouse in B2B startup
25mData Demons: Unleashing Chaos or Driving Performance?
31mExploring open data with BigQuery
41mAnalyzing Terabytes of Data with Google BigQuery
17mCloud Compute Cost Showdown