Debugging Clojure Services Under Stress
summary
Three-step debugging: identify the unknown, act to restore service, then improve to close the loop. A real incident traces a backed-up batch queue to a new slow DB query run on a replica, reproduced there, hotfixed in ~10 minutes and permanently patched the next day. Practices include continuous deploys with quick rollbacks, a checked-in diagnostic namespace deployed to prod, logging to search plus Sentry and OfficeGenie alerts, use the REPL, sketch architecture with plantuml, beware accidental edits like changing 25,000 records instead of 200.
AI-written summary · about
chapters (21)
- 0:00Intro
- 2:20WHY IS DEBUGGING PRODUCTION ISSUES STRESSFUL?
- 2:56STEP 1: MENTAL PROCESS OF DEBUGGING
- 3:45DEBUGGING PROCESS
- 5:27MY DIAGNOSTIC STEPS FOR PRODUCTION ISSUES
- 7:26DEBUGGING WITH CLOJURE
- 8:01MY COPING STRATEGIES
- 9:18NEGATIVE THREAT
- 10:00GET PREPARED Pakistan
- 12:22RECORD DEBUGGING STEPS
- 13:10DEVELOP A DEBUGGING PLAYBOOK scope problem - control flow diagrams, UMLS
- 14:52DEBUGGING WITH JUPYTER NOTEBOOK
- 15:08DOCUMENT CONTROL FLOW
- 17:00GETTING YOUR SYSTEM PREPARED TO MAKE YOUR LIFE EASIER
- 17:23CONTINUOUS DEPLOYMENT + ROLLBACK CAPABILITY
- 18:44DECOUPLE BUSINESS LIFECYCLES
- 19:43ROUTE ERRORS BY LEVELS
- 21:54ACTIONABLE ERROR MESSAGE
- 22:36DESIGN FOR FAILURES
- 23:04CLEAN CODE
- 24:00BASICALLY, PRACTICE GOOD SOFTWARE ENGINEERING
related talks
34mChaos Debugging: Finding and Fixing Microservice Weak Points
39mResolving Outages Faster with Better Debugging Strategies
31mDebugging Techniques in Elixir
28mResolving Outages Faster With Better Debugging Strategies
15mDebugging debugging; why we must approach debugging differently
26mThe Mindset of Debugging
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures
26mow to Enjoy Debugging in Production with Karen Tracey