Scaling a Distributed Container Management System: Diego Performance
summary
Diego runs container workloads at scale and the team measures that with concrete experiments: a smoke test boots 4,000 instances in <30s, a full run saturates 100 cells / 10,000 instances with BBS API latency under 1s and no long-term degradation, and failure drills show cells rebalance missing workloads within ~30s and a full DB restore repopulates desired state in ~10–15 minutes. Tests use garden-linux, logger-gator + DataDog, protobufs and RPC; etcd v2 bottleneck leads to a relational backend (MySQL/Postgres) that stores 600k+ instances in tests.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:31What is Diego
- 5:10Scaling Goals
- 7:07Measuring Performance
- 8:31CFC Internal Logging
- 8:58CFC Internal System Logging
- 9:49Smoke Test
- 12:46Time to Scale
- 14:10Experiment Design
- 16:03Experiment Results
- 16:49Failure Modes
- 17:41Total Database Failure
- 18:22Cell Failure
- 19:23Performance Targets
- 19:57PBS
- 20:53Results
- 21:49Scaling with Blockers
- 24:40Out of Experimental Mode
- 24:59Memory Caching
- 26:19Questions Answers
related talks
37mCloud Foundry Diego Overview
48mScale and Performance Testing of Kubernetes: Answers for Specific Applications
30mBuilding distributed systems is hard - case study (...)
29mIntro + DeepDive: SIG Scalability
32mImplementing the Schedulers: A Look at the Internals of Diego...
43mScalability Intro – Shyam Jeedigunta, Google & Bob Wise, AWS (Any Skill Level)
24mTuning docker network for large scale production workload
43mLinux Container Performance Analysis