Building Large Scale Services
summary
Built a globally replicated, eventually consistent key‑value service named Sherpa and iteratively hardens it by writing automation and processes rather than just docs. Key specifics: Sherpa: globally replicated eventually-consistent key-value store, 100+ automation tools created over five years, 500+ operability bugs filed; early on operations is 24/7 with one person on call and later shifts to ~75% tooling work. Uses Chef/Puppet, checklists, fire‑drills and a “give me the brain” docs site; failure exposes hidden process gaps.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 2:17System admin outside the stack
- 3:43Stop treating users as stupid
- 5:50Operability and character sheets
- 7:16Visualizations reduce fake work
- 9:03Teams as roles
- 10:47Vision statements matter
- 12:55Understand platform value
- 14:00Big operability issues
- 15:24Effective checklists
- 16:51Avoid distributed assumptions
- 17:56State transitions in service
- 19:20Fire drills over postmortems
- 21:05Success needs teamwork
- 23:35Hidden failures in success
- 25:43Plan exit from day one
- 26:47Q&A
- 27:29Team player tips
- 28:55Time management growth
- 31:07Gamification in operations
- 33:59Documentation challenges
- 36:07Automate operations tools
- 38:35Final thoughts
related talks
42mPlanning To Fail
01h 02mTools for simplified large scale systems
19mEngineering for the Long Game
31mDevOps in the Cloud: Case Studies of Amazon.com teams and their resilient architectures
32mScaling Done Right
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
33mScaling Chef Emotionally
47mDeploying Like a Boss: From 2 Servers to 20,000