Challenges Faced and Lessons Learnt While Building a Multi-cloud Data Platform
summary
Transforms an AWS‑centric data platform into a multicloud setup, implements a Hive proxy catalog that mimics Glue and a custom Spark catalog, and runs workloads via Spark on Kubernetes (EKS/AKS) while keeping EMR and Glue for compatibility. CI/CD hides S3/ADLS details and adds deployment descriptors. Six months in only 5% multicloud adoption; 85% of users run the new framework; 12% of new apps target Azure and 3% of all apps run on Azure. Caching, smart scheduling, and true independence remain unresolved.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:54Moving to multicloud data platform
- 13:47Multicloud meaning and value
- 16:04First multicloud challenges
- 18:57Choosing compute engines
- 21:26Hiding infrastructure details
- 26:46Platform architecture overview
- 28:38Gentle rollout and results
- 30:30Lessons learned
- 31:55Next steps and improvements
related talks
39mExperiences and Lessons Learned Building a Multi-Cloud Spring Boot Backend
57mCloud Adoption Patterns
15mJourney to Hybrid Multi-Cloud
24mCICD Pipeline and delivery of Apache Spark Applications on the cloud using AWS
43mGetting to a Multi-Cloud Deployment - A Case Study of
35mKubernetes journey
38m50 Shades of Multi-Cloud Infrastructure
34mLessons Learned from Five Years of Multi-Cloud