Lessons scaling operations to everyone @indix
summary
Indix moves from one ops person and a centralised chef server to teams owning infrastructure with Ansible, Marathon/Mesos and GoCD; engineering headcount grows from ~5–15 to 30–50 while ops stays ~1–3. Claims: decentralisation reduces midnight pager helplessness but drives cost spikes ($150k→$220k/month) and fragmentation. Concrete fixes: per-team playbooks, operability review checklist before production, DNS backups every 30 minutes after a production overwrite, and a tech radar to standardise tooling.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:05Three stages
- 1:19Early stage
- 4:56Devs
- 6:09Growth
- 8:25Cost
- 10:58Later stages
- 12:39Microsoft schedulers
- 13:54Tech Radar
- 15:06Operability Review
- 16:10Recap
related talks
52mDevops @Scale
48mDevops@Scale
55mDevOps @Scale (Greek Tragedy in 3 Acts)
58mDevOps @ Scale — Greek Tragedy in Three Acts
36mDevOpsSup - Extended DevOps approach
18mScaling DevOps in Complex Environments: Challenges, Failures, and Successes
57mGo Away Or I Will Replace You With A Very Little Shell Script
22mRaise a Production System:Making on-call human-scale w/ Observability & Trust
