Architectural patterns of resilient distributed systems
summary
Argues that resilience is decided by concrete choices not buzzwords and shows literature and industry evidence for that claim including a 66% harvest example, Google opting for a centralized lock service, and Netflix moving from chaos monkey to real time traffic visualization. Describes tools actually built and used at Fastly: a working instant purge system called Powder Horn implemented with a gossip protocol and protocol hacks, and explains when those approaches succeed, what failures remain, and the operational costs of keeping them running.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:18Architectural Patterns of Resilient Distributed Systems
- 2:03Resilience is the ability of a system to adapt or keep working when challenges occur
- 8:12Graceful harvest degradation under faults Randomness to make the worst-case & average-case the same Replication of high-priority data for greater harvest control Degrading results based on client capability
- 9:30#2 Decomposition & Orthogonality Decomposing into subsystems independently intolerant to harvest degradation but your application can continue if they fail Only provide strong consistency for the subsystems that need it Orthogonal mechanisms state vs functionality
- 13:46Engineering resilience requires a model of safety based on: mentoring responding, adapting, and learning System safety is about what can happen, where the operating point actually is and what we do under pressure
- 14:43Engineering system resilience Build support for continuous maintenance Reveal control of system to operators know it's going to get moved replaced, and used in ways you did not intend Think about configurations as interfaces
- 15:05Engineering system resilience Build support for continuous maintenance Reveal control of system to operators know it's going to get moved, replaced, and used in ways you did not intend Think about configurations as interfaces
- 20:21Thinking about building system resilience using a single discipline is insufficient. We need different strategies 11
- 25:08fastly. Powderhorn insights Evolution of our purging system from vi to v3 Used Bimodal Multicast Gossip protocol to provide extremely fast
- 27:07Image Opto insights A stateless system is nice but can be tricky Dependencies are hard: customer setup, caching layer, & libraries - we have to be resilient to all of them
- 29:45Design error types & their handling carefully Failure detection & system operability are ongoing concerns Mixed-mode & versioning of data structures Validation & system adaptability
- 31:44Redundancies are key Redundancies of resources Execution paths. Checks replication of data replay of messages, anti-entropy build resilience Gossip/epidemic protocols Capacity planning matters
- 33:01Operations matter Unawareness of proximity to error boundary means we are always guessing Complex operations make systems less resilient & more incident.prone You design operability too!
- 42:08Good design is hard. Unknowns are hard to predict. Let the tenets we discussed today guide your redesigns - Me today
related talks
51mPatterns for building resilient software systems
01h 05mKeynote - Without Resilience, Nothing Else Matters
16mResilient by Design: Architectures for Crisis Response & Recovery
51mWithout Resilience, Nothing Else Matters
53mResiliency and Availability Design Patterns for the Cloud
45mFast, Ever-Evolving Defenders: The Resilience Revolution
32mResiliency and Availability Design Patterns for the Cloud
16mEngineering Failure-Resilient Systems