Hodor: Detecting and Addressing Overload in LinkedIn Microservices
summary
Hodor is a framework deployed to close to a thousand linked services and engaged hundreds of times in production to detect overloads and shed traffic automatically without per-service tuning. It uses three concrete detectors — heartbeat (thread scheduling delays), GC percent-time and Jetty thread‑pool wait — and adaptive shedding strategies (percent, concurrency, throughput with traffic tiering). Retry budgets prevent storms, rollout uses monitoring‑only waves, false positives drop by requiring latency correlation, and the system proves effective in practice.
AI-written summary · about
chapters (22)
- 0:00Start
- 2:21Defining What an Overload Is
- 2:57Physical Resources and Virtual Resources
- 5:05Examples of Overload Scenarios
- 5:16Increase in Traffic Can Push a Service into Overload
- 6:50Mitigating Overloads
- 7:32A Noisy Neighbor
- 9:55Load Shedding Strategy
- 12:07Cpu Detector
- 14:57Garbage Collection Detector
- 16:40Metrics Collected during a Gc Overload
- 19:02Metrics from a Thread Pull over Overload Event
- 19:44Load Shedding Strategies
- 19:54Adaptive Percentage
- 21:35Limiting the Number of Concurrent Requests
- 23:25What Happens to the Requests That Are Dropped
- 24:59Traffic Tiering
- 25:18Traffic Tiers
- 31:55Data Analysis
- 33:02Roadmap Related to Hodor
- 40:33How Do You Know When To Stop Shedding Traffic
- 42:05Does Load Shedding Create Back Pressure for Upstreams
related talks
51mBuilding Fault Tolerant Microservices
44mDeep Dive talk | Resilience in Microservices with Congestion Control
29mFault Tolerance is a Requirement Not a Feature - Adar Danait, Lilian Ernest, Citi
53mHow do we build high‑load microservices
49mThe modernization of legacy systems with microservices
40mInfinitely scalable microservices with Project Loom and Helidon
19mNodens: Enabling Resource Efficient and Fast QoS Recovery of Dynamic Microservice...
45mAvoiding Microservice Megadisasters