How Big is Too Big? Problems with Scale at T-Mobile
summary
T‑Mobile runs Cloud Foundry and PKS on‑premise and hits scale limits: a largest foundation holds 16,000 apps, 365 orgs, 2,200 spaces and ~43,000 service bindings, logging 4.5 billion messages/day and dropping 5–10%. Operators see TCP buffer loss, firehose disconnects every minute or two, stemcell repaves that take 18–24 hours, and unpredictable on‑demand service upgrades. Response is horizontal scale — many smaller foundations — plus per‑foundation Concourse, an OSB geo‑load broker with F5 GTM DNS routing, and heavy automation work.
AI-written summary · about
chapters (17)
- 0:00Introduction and team overview
- 1:20Operational improvements
- 2:42Vertical scaling challenges
- 4:16Infrastructure and upgrade woes
- 7:01Blast radius and API latency
- 8:53Logging and capacity issues
- 10:02Horizontal scaling complexity
- 12:45Geo-load balancing solutions
- 14:01Community feedback
- 15:06Q&A: Global load balancing
- 16:59Q&A: Multi-foundation management
- 18:50Q&A: CI/CD and automation
- 20:34Q&A: Platform-user relationship
- 23:11Q&A: Infrastructure and staffing
- 25:01Q&A: Log management strategies
- 27:09Q&A: Feature rollout and selectivity
- 29:28Q&A: Upgrade processes and closing
related talks
40mThe Art of Embracing Failure at Scale
27mCloud Foundry in Production
26mObservability & Monitoring in Cloud Native Telco World – Tying It Together
26mGrowing Pain: Scaling to 10K Workflows per Week
35mKubernetes, Lessons Learnt
32mTuning and Managing Loggregator, the Cloud Foundry Logging Subsystem
20mHow Can You Scale It If You Don't Trust It?
05mPreventing cascading failures in a global network