How to Write a Service Whose Maintenance Doesn't Turn into Hell
summary
Covers developing a service with operations in mind for large-scale systems (about 1,500 machines and millions of users). Explains practical logging practices, techniques for localizing and reproducing incidents, and approaches to incident investigation so debugging doesn’t become a time-consuming search through noisy logs.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:44Goals and format
- 1:08Why write logs
- 3:22Log evolution over 10 years
- 4:14Case 1: File data loss
- 6:01Case 2: User creation errors
- 9:58Improving logs: Details and context
- 10:20Case 3: Slow query
- 11:27Case 4: Large file transfer
- 14:20Statistics for analysis
- 15:25Case 5: Video conversion errors
- 18:42Importance of log separation
- 19:24Case 6: Video playback errors
- 20:49Logging rules
- 21:55Log size and privacy
- 22:16Timeout death query
- 24:13Tracking user requests
- 25:59Summary
- 27:54Q&A
related talks
10mBuilding an 'Oops-Less Operation' at Scale
38mBuilding Service Ownership Using Documentation, Telemetry, and a Chance to Make
32mScaling Done Right
24mThe Lifecycle of A Service
45mToward a Culture Where Everyone Is Responsible for Outages
26mSustaining Engineering - life after devops?
15mMaintaining reliable systems
29mTaking Over & Managing Large Messy Systems