Tuning docker network for large scale production workload
summary
Runs Docker at 250→1000+ Docker daemons and sees Docker-network ops go from seconds to minutes as etcd CPU and disk IOPS saturate. Identifies Libkv inefficiency and outdated Docker version, upgrades Docker, moves etcd onto SSD-backed machines with 16 CPU / 32 GB, increases docker heartbeat from 20s to 1000s to cut watch-chatter, and finally replaces a single shared etcd with dedicated per-cluster etcd, restoring fast container startup and predictable scale for thousands of hosts.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 3:12Software-defined networking with etcd
- 5:02First performance degradation issue
- 8:24Docker Libkv bottleneck
- 10:41etcd hardware limitations
- 12:30Hardware failure and assumptions
- 14:20Scalability testing at 400 demons
- 17:42Heartbeat and network chatter
- 19:33Dedicated etcd per cluster
- 22:28Key lessons and conclusions
related talks
31mScaling a Distributed Container Management System: Diego Performance
13mPerformance and Energy Efficiency in Docker
48mScale and Performance Testing of Kubernetes: Answers for Specific Applications
43mLinux Container Performance Analysis
34mLoad Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production
39mStaying in Tune: Optimize Kubernetes for Stability and Utilization
15mDemo: Resource(Cost) optimization on planet scale Kubernetes clusters.
43mComparing Performance of NVMe Hard Drives in KVM, Baremetal, and Docker