Getting back to sleep as soon as possible for the on-call developer
summary
Middle-of-the-night payment failures trace to a microservice bug that hides from logs: filtering by 24085 user id shows no errors, but source inspection finds an auto cache left by an IDE that breaks flow. Recommended fixes are concrete: add targeted logs, change log levels, use production debuggers/continuous observability (LightRun mentioned), keep runbooks and shell scripts ready, run postmortems — a profiler example cuts a 7-minute query to 7 minutes to 7 seconds by adding a cache.
AI-written summary · about
chapters (15)
- 0:00Introduction to on-call sleep
- 1:55Speaker background and talk origin
- 2:37Real-life vs. ideal system problems
- 4:44Understanding observability basics
- 8:59Chaos engineering and edge cases
- 14:54Typical on-call company setup
- 18:46Demo of on-call debugging process
- 22:56Limitations of current debugging tools
- 23:37Continuous observability solutions
- 24:42Strategies for winning at on-call
- 26:50Prepare tools and runbooks
- 29:19Automate visibility with scripts
- 30:23Post-mortem and improvement
- 32:09Team communication and collaboration
- 35:18Closing remarks and resources
