Migrating Our Data Warehouse to the Cloud
summary
Moves 1.5–2 TB of warehouse data from on‑prem into Google BigQuery and AWS RDS, replaces Azkaban with Airflow and breaks a monolithic pipeline into a 1,000-task daily DAG split into staging/facts/analytics/cubes, builds SSAS on EC2 and automates provisioning with CloudFormation and AMIs, uses BCP/SSIS for bulk loads and Apache Beam validation to compare hashes, migrates about 30 GB of Excel spreadsheets for ~300 users, finishes with stable runs after initial issues and half the ongoing ops effort.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:23About the team
- 0:37About me
- 1:09Data Center Architecture
- 3:59Migration Steps
- 5:15Azkaban vs Airflow
- 6:51What do we need to migrate?
- 7:39Big daily process
- 8:23Identifying and splitting up
- 9:03Introducing checkpoints
- 10:55Checkpoints approach
- 11:23Ignoring the black box
- 12:06Old code
- 13:12Advantages
- 14:36SQL Server
- 18:48Summary
- 21:10OLAP Server
- 29:12Automated validation
- 32:00Custom validation
- 32:52What we learned
- 34:57Timeline
related talks
47mData Warehousing Migrations: Lessons from Home Depot
45mBuilding a Data Warehouse
36mThe MODERN Modern Data Stack: Building an Open Distributed Data Warehouse Beyond...
41mLessons Learned from the GCP Data Migration
58mTales of Data Architecture Evolution
28mModernize Your Data Warehouse
37mBuilding a Data Platform from scratch
25mData Demons: Unleashing Chaos or Driving Performance?