Containerizing & Orchestrating Hadoop Infrastructure at Scale
Talkat ContainerCon @ Open Source Summit Europe 2022 · on Friday, 2022-09-16· at 13:55· Wicklow Hall 2A (Level 2), Convention Centre Dublin, Spencer Dock, N Wall Quay, North Wall Dublin 1, D01 T1·
summary
Uber moves its Hadoop stack into Docker and an internal orchestrator, running ~100,000 steady containers and launching 20 million JVM processes per day. A Starlark-based config generator (≈4 KB) expands into ~150,000 lines of cluster XML, letting templates and runtime vars handle per-cluster differences. Node onboarding uses an Aspire attestation agent and automated secret provisioning. Migration uses reconciliation loops, automated decommission workflows that migrate ~75% in a year, and a zero-downtime rollout strategy.
AI-written summary · about
chapters (7)
- 0:00Start
- 2:04Background
- 8:02Challenges
- 19:44Engineering the Migration
- 28:45The Cultural Shift
- 31:03Key Takeaways
- 31:33Future
related talks
24mDynamic Big Data in the Cloud with Kubernetes
46mContaining an Elephant: How we moved Hadoop/HBase into Kubernetes and Public Cloud
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
28mDevOps in a Data Warehouse: Inside Out
45mHadoop in the Cloud: The Story of Migrating Hundreds of Petabytes
31mA Journey of Migrating Data Platform from Data Center to Cloud
34mFrom Data Centers to Cloud Native - Dave Zolotusky & James Wen, Spotify
47mAll the DataOps, all the paradigms