Taking Over & Managing Large Messy Systems
summary
A managed‑service team in China documents concrete tactics for taking over live messy systems serving 1M–300M users across 50–100 datacenters: emergency stabilization within about a week, deep config inspection with a collector that emits JSON into a CMDB, rule engines with 500–1,000 rules that flag bad MySQL and kernel settings, centralized logs in Elasticsearch, and deployments using Ansible, Zabbix for monitoring and HAProxy for rolling upgrades. Many sites still run decades‑old OSes, broken backups and multiple Tomcats; the tooling and phased playbook largely succeed at stabilizing without downtime.
AI-written summary · about
chapters (28)
- 0:00Intro
- 0:42Managed Service Provider
- 3:59Have to Rebuild the Racecar During the Race
- 5:38Auto-Install Monitoring & Logging
- 6:55SRE Golden Signals
- 7:43Deep CMDB
- 8:23OS & Service Audit
- 8:51Cloud Audit & Governance
- 9:24Find all the bodies
- 10:43Amazing Apaches
- 11:12Microservice Madness Makes it Messier
- 11:55Link & Dependency Discovery
- 12:38Deep Monitoring
- 13:27Logs & Log Metrics
- 14:20Backup Checkers
- 15:21File Permission Digger
- 16:24Buried Body Discovery Phase Ends
- 16:353 Rebuild the Race Car during the Race
- 17:38Tools to Rebuild
- 18:45Automation
- 19:17Load Balancers & HA
- 20:35Examples
- 20:51Manage & Optimize
- 21:23Requirements
- 22:26Watch 'em like a Hawk
- 22:46Long-Term Ops Tools
- 23:01Lessons Learned
- 23:30Take-Aways
related talks
47mWhy keeping your system secure is a bit like decluttering
42mCatastrophes and Atrophies
01h 02mTools for simplified large scale systems
41mLessons from Mission-Critical Systems in Fast-Moving Industries
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
55mDevOps @Scale (Greek Tragedy in 3 Acts)
32mThe Necessity of Configuration and System Management Tools
23mHow to tame chaos effectively?