Debugging Large Scale Service Oriented Systems
summary
Proposes instrumenting every incoming connection with a request ID stored in a thread local, logging start/stop times and attaching the ID to exceptions, Resque jobs (via Resque onq/around hooks) and HTTP client calls (RatClient middleware), then indexing everything in Elasticsearch for cross-service traces. Claims this yields full request traces, correlates duplicate errors across frontends/backends, and exposes which service or process fails. Implementation is prototype-level and conceptually tested at Engine Yard but not yet fully deployed to production.
AI-written summary · about
chapters (10)
- 0:00The debugging challenge
- 2:27Concept of request tracking
- 3:52Generating request IDs
- 6:59Logging requests
- 8:29Error correlation
- 10:08Background job instrumentation
- 15:59Client-side middleware
- 19:40Benefits of request traces
- 22:55Data storage solutions
- 26:25The Google Dapper model
related talks
31mTroubleshoot Your RoR Microservices with Distributed Tracing
30mService Oriented Design in Practice
23mAbstracting a Service - A Case Study in Forward Thinking with SOA
36mAn Iterative Approach to Service Oriented Architecture
54mServices, Services, Everywhere!
04mObservability in micro-service architectures
31mUnderstanding Microservices with Distributed Tracing
33mEffective observability in microservice architectures