Expect the Un-expected: How to Handle Errors Gracefully
summary
ResearchGate runs 14 million users at about 1800 requests/sec across ~140 services and 400 repos with ~50 deployments/day. Logging ships to a central ELK pipeline (Filebeat/Logstash → ElasticSearch → Kibana) with correlation IDs and file fallback; metrics go to Graphite via statsd and alerts through Nagios/PagerDuty. Feature flags enable gradual rollouts; componentized front-end degrades missing pieces; circuit breakers (Hystrix or via linkerd) prevent cascades. These measures cut incident blast radius and measurably improve availability and performance.
AI-written summary · about
chapters (34)
- 0:00Intro
- 0:39140 components
- 0:50Active development
- 1:15Performance regressions
- 1:49How to detect
- 2:33Error Logs
- 3:22Request Information
- 3:58Log in a structured way
- 5:20Correlation / Tracing ID
- 7:09Full text search
- 9:49Always Log to file
- 10:25Conversion
- 10:53Latency
- 11:23Granular
- 11:39Graphite
- 12:25Prometheus
- 12:41Dashboards
- 15:37Handling Exceptions gracefully
- 15:54Component based fronted
- 16:34Degrading Functionality
- 18:04Big cause for regressions
- 18:19Safe deployments
- 19:52Fast rollbacks
- 20:49Safe Service-to Service Communication
- 21:04Distributed Systems are hard
- 21:30Very slow responses
- 22:24Low timeout settings
- 22:54High availability
- 23:32Health checks
- 23:51Circuit Breakers
- 27:10Handling traffic spikes
- 31:51Central Log Management
- 32:14Monitor and Measure everything
- 32:34Graceful Exception Handling
related talks
42mPlanning To Fail
45mEngineering resiliency - how I learned to love an outage
42mMake your logs work for you
39mOops. I broke the Google, now what?
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
34mMetrics driven development
32mExpect the Unexpected: How to Deal with Errors in Large Web Applications
41mScaling Observability: From Chaos to Clarity with Open Source Tools