"Ops War Stories" Lightning talks
summary
Lightning round of operator war stories focuses on concrete fixes and surprising scales: a routed Calico OpenStack cloud hits incompatible BGP and Skylake TSC drift, triggers a rolling kexec firmware-and-kernel patch that live-evacs 1500 hypervisors and ~15,000 VMs using nova auto-converge and one-to-one rack swaps; bad admin filters once deleted all volumes or returned every Neutron port; an apt cron at 02:00 causes daily test failures; kernel differences freeze router namespaces during mass node replacement.
AI-written summary · about
chapters (9)
- 0:00Introduction and setup
- 9:04Session overview and rules
- 14:22War story one data center collapse
- 21:55War story two volume deletion mistake
- 27:39War story three cron job at scale
- 33:21War story four hardware vendor bug
- 39:18War story five ephemeral storage cleanup
- 45:26Future of ops meetups and ops radio
- 51:30Q&A and closing remarks
related talks
05mMike Jang - Ignite OSCON 2014
06mIgnite: Moving from regular sysadmin to DevOps
34mFrom a Backend Developer to DevOps - THE FORCE AWAKENS
32mA Day in the Life of a Kubernetes Engineer
32mMy Great Awakening: Top “Aha” Moments As Former Dev Manager
39mThe Beginning, Present, and Future of Sysadmins
29m"Engineering! I need more power!"
29mFrom Sysadmins to (almost) Flying Unicorns