Migrating a data stack from AWS to Azure #bbuzz
summary
Migrates a full AWS data stack to Azure by replacing Hadoop/EMR and Redshift-centered ETL with Unix tooling: make plus Python, a homegrown Junior Beaver mapper, and blob storage instead of S3. Benchmarks show one 4‑vCPU node (and a Raspberry Pi testbed) handles what formerly took hundreds of vCPUs, reaching 6 million events per minute on bigger hardware and Raspberry Pi prototype at ~1 million/min. Azure SQL Data Warehouse and blob access prove functionally comparable and cost-competitive after tuning.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 0:36The Metatron
- 2:32How big we are
- 3:12microservices
- 4:20data
- 5:02data flow
- 7:08ETL system
- 8:50Redshift
- 10:51State of Union
- 12:14bucolic data landscape
- 13:36Make it simple
- 15:00Big run kills big data
- 19:46Crazy Idea
- 21:32Fix Up
- 22:08Azure Blob Storage
- 23:18Azure Data Warehouse
- 25:37Buzzards
- 28:08Lazy and paranoid
- 28:58Measuring beans
- 29:48Fairy
- 30:37Questions
- 33:39Resources
related talks
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
18mMigrating my favorite past project from On Premise to Azure
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
32mMove to managed analytics
54mAzure Best Practices for Lift and Shift
08menterprise data culture
50mMigrating data solutions to the cloud - a checklist.
28mFlexible Cloud, Whatever The Workload