Ops issues
summary
Operators report many clusters still on Mitaka and Rocky so upgrades require sequential migrations (A→B→C), not skip-level jumps. Third-party plugins (Calico, Contrail/Tungsten) and vendor CI lag block upgrades. A recent change turns on auto-converge and tunes live‑migration timeouts to reboot a fleet of 1500 hypervisors, moving ~20,000 VMs successfully. RabbitMQ scale fixes include setting mirrored queues to exactly two replicas. Prometheus setups often scrape multiple instances behind a load balancer, trading durability for simplicity.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:30Etherpad
- 4:00Openstack
- 8:00Superuser nomination
- 11:00Upgrade cadence
- 13:00Upgrade pain
- 17:00Contrail
- 20:10Certification
- 22:25Certification vs CI
- 23:55Driver support
- 26:00Realignment
- 29:15Live Migration
- 33:35Rabbit
- 35:15RabbitMQ
- 37:35Inconsistency
- 43:35Thank you
- 45:10Future Ops meetups
related talks
46mEnd-Users Won't Even Notice, Painless Kubernetes Upgrades
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
28mDesigning for Failure: How to Manage Thousands of Hosts Through Automation
35mHandling Risky Business: Cluster Upgrades
15mKeeping a Hypervisor Fleet Up to Date with Minimal Customer Disruption
06mNever ever upgrade
32mDon't Start With 500 Clusters: Patterns in Scaling Multi-Cluster Kubernetes…
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)