Billion-Scale Triumph: Innovating at the Edge of Cloud Data
summary
Migrates a 3+ PB on‑prem Cloudera system to cloud using Databricks autoloaders and Azure SQL Hyperscale, ingesting a baseline ~4 billion records/day and peaking to tens of billions after optimizations. Builds Terraformed infra, Databricks‑SDK orchestration, custom connectors and client‑specific auth, and a bespoke monitoring stack to track tuple‑mover/metadata. Achieves ~60% cost reduction and up to ~25M rows/min throughput in private preview, but clustered columnstore metadata and tuple‑mover behavior cause multi‑hour outages and force a partial architectural redesign.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:12Databricks and Azure SQL Hyperscale choice
- 7:26Azure SQL Hyperscale architecture
- 11:00Technology validation and performance
- 15:19Optimizing data ingestion speed
- 19:50PySpark repartitioning challenges
- 25:56Bulk inserts and columnstore index
- 30:17Metadata issues and monitoring
- 32:43Partitioning challenges and outage
- 33:46Project status and lessons
- 34:48Q&A
related talks
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
40mProductionizing Big Data - Stories from the Trenches
28mFlexible Cloud, Whatever The Workload
57mScaling Databricks to Run Data & AI Workloads on Millions of VMs
39mMigrating a data stack from AWS to Azure #bbuzz
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
01h 03mDiscovering the value of a cloud data platform
44mData Mesh: Lessons from the trenches