Architecting for Failures in Microservices

on YouTube ↗42m

" summary

Distributed microservice outages are traced to concrete causes and fixes: a morning surge of 500 errors exposed slow API latency that crept up over time, a bug added an ORDER BY that caused a full table scan (queries rose to ~718 ms average and ~2 s 95th), and a 24‑hour cache TTL exhausted validation codes until reduced to 12 hours. Remedies include Hystrix circuit breakers, fallbacks and predictive caching, bulkheads, retries with exponential backoff and jitter, Zipkin tracing and PagerDuty alerts plus failure injection, load tests, runbooks and phased rollouts.

AI-written summary · about

" chapters (14)

1. 0:00Start
2. 5:00PATH TO MICROSERVICES
3. 7:26RESILIENT SYSTEM
4. 16:08LESSONS LEARNED
5. 16:43ACTION PLAN
6. 18:46RESILIENCY PLANNING STAGE 1
7. 19:51CASCADING FAILURES
8. 22:09BULKHEAD
9. 25:40TALE OF THE NEVER LIVING CACHE ENTRIES
10. 26:49LOEGING BEST PRACTICES
11. 29:48CAPACITY PLANNINE
12. 30:32RESILIENCE PLANNING STAGE
13. 30:48HEALTH CHECK
14. 35:57ROLLOUT OF NEW FEATURES

" related talks

• 45mAvoiding Microservice MegadisastersJimmy Bogard · NDC London 2017
• 46mSurviving microservicesFrancesco Degrassi · IDI Incontro DevOps Italia 2020
• 39mTop Performance Challenges in Distributed ArchitecturesAndreas Grabner · Confitura 2019
• 51mBuilding Fault Tolerant MicroservicesKristoffer Erlandsson · Jfokus 2016
• 39m10 Tips for failing at microservicesDavid Schmitz · code.talks 2017
• 23mMicroservices – Diptanu ChoudhuryDiptanu Gon Choudhury · Microservices Day NY 2015
• 57mEmbracing MicroservicesGrzegorz Piwowarek · GeeCON Prague 2023
• 10mBuilding an 'Oops-Less Operation' at ScaleChris Engelbert · JJUG CCC 2019 Fall