Share your OpenStack scaling story
summary
Operators report that neutron and its agents are the first bottleneck, causing cascading failures into rabbitmq overload and databases; typical hard limits appear around ~1,000 nodes per region while some sites split into regions/cells to survive. Fixes that actually work: increase RPC worker counts and timeouts, tune rabbitmq and MariaDB/Galera, use conductor groups for ironic, enable cinder trash, throttle agent restarts, or shard control planes. Surprising failure modes include memcached socket timeouts, open-file limits, HAProxy connection timeouts, and deadlocks when creating 200–300 ports in parallel.
AI-written summary · about
chapters (9)
- 0:00Intro
- 4:08OVH
- 9:51RabbitMQ
- 12:51Cern
- 19:59Cinder issues
- 21:40Keystone issues
- 26:00API issues
- 29:15A point scaling exercise
- 35:52Parallel vms
related talks
52mLet’s build a private cloud – how hard can it be?
19mKeynote: Operating 600+ Mesos Servers on 7 Datacenters @Criteo
43mSupporting NASA Science with OpenStack
29mKubernetes Failure Stories and How to Crash Your Clusters
10mNever Go Limitless – On the Importance of ResourceQuotas at the Edge
20mManage Scaling, While Keeping Costs Under Control. Challenge Accepted.
44mHow we designed an OpenStack Public Cloud at Scale with Fabric Networking
46mScaling a Collaboration Service like Nextcloud to 20 Million users