How to Use Common Investigation Practices in Troubleshooting
summary
Claims a detective process speeds mean time to repair by applying concrete steps: use Gatus or a simple health checker instead of heavy VictoriaMetrics for small stacks, keep monitoring external to the primary server and split alerts by channel and owner. Describes tools and tactics used: Bombardier or remote-access workarounds, HAProxy diagnosis, journalctl and ncdu for evidence, reproducing on stage and rollback with Git and Terraform, and a Tableau case where isolating HAProxy to network pinch points fixes production.
AI-written summary · about
chapters (20)
- 0:00Troubleshooting in DevOps intro
- 1:33Troubleshooting beyond hard skills
- 2:55Lessons from detectives
- 5:24Three-step investigation
- 6:28Incident reports and monitoring
- 9:58Alert management
- 11:47Urgency vs importance
- 12:51Accessing infrastructure
- 15:19Hot pursuit under pressure
- 16:47Time management and escalation
- 18:13Occam’s razor in troubleshooting
- 19:18Interviewing witnesses
- 21:22Evidence collection
- 24:09Building hypotheses
- 26:19Resolving Tableau issue
- 29:35Timelines and post-incident
- 30:38Reproducing on staging
- 32:03Root cause analysis
- 33:27Learning from mistakes
- 34:33Q&A on AI tools
related talks
43mProd Detectives: Solving the Case Before the Crash
24mGremlins Exposed: Shining a Light on Mischievous Systems
40mProduction Detectives: Solve the Case Before the Crash
01h 05mThe art and methodology of troubleshooting
39mResolving Outages Faster with Better Debugging Strategies
38mLost Art of Troubleshooting
41mDetective Hat: Investigating Production Issues
21mLessons From Long-Running Investigations