Data Warehouse in Google Cloud
summary
Toptal builds a Data Warehouse on Google Cloud using BigQuery for self-service analytics, Google Cloud Storage for daily Avro snapshots, and an ETL pipeline in Python/Scala managed by Luigi and Avro; two to three engineers deliver a production prototype in a couple months (≈200 handwritten LOC, 500 total) and run it for a year as a scalable reliable ETL. App Engine microservices expose ML and cached Redis views; practical fixes include generous retries; tradeoffs are query latency and coarse BigQuery permission granularity.
AI-written summary · about
chapters (29)
- 0:00Intro
- 1:37Why Im here
- 2:08Agenda
- 2:38About Cocktail
- 3:47Analytics Department
- 4:25Why the Warehouse
- 4:50Initial Idea
- 5:30Spark Scala
- 5:54Execution History
- 6:25monolithic
- 6:47SQL
- 8:02Data Warehouse
- 8:20Data Pipeline
- 9:15Technologies
- 9:41Cloud Storage
- 10:43BigQuery Analytics
- 12:09File format
- 12:44Data types
- 13:15ETL
- 14:00Target
- 14:45Microservice
- 15:23Example
- 23:33Retry Google interactions
- 26:18Production ready
- 26:37Prototype
- 26:55Integration
- 28:12Reliability
- 28:45High Latency
- 29:21Permissions
related talks
26mETL pipeline with GCP Dataflow
47mData Warehousing Migrations: Lessons from Home Depot
43mData warehousing reinvented for today's needs
44mBig Data Science on Google Cloud
33mBigQuery DeepDive: dissecting Google's Data Warehouse
30mA Journey to Choose an Enterprise Big Data Platform
15mData strategy: Getting ready for Cloud migrations
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery