Improving Availability in Distributed Systems with Failure Informers
summary
Pigeon implements a failure informer service that groups low-level signals into actionable classes (examples: stop vs unreachability and resource depletion) and reports expected duration so applications can take tailored recoveries. It uses embedded sensors and relays plus a timeout backstop; prototype is ~5k LOC, runs on a 16-router/12-host testbed, and integrates with Cassandra (400 LOC) and RAMCloud (70 LOC). Results: primary/backup downtime falls from 6.9s to 1.6s, avoids costly RAMCloud recoveries, CPU overhead ≈3% hosts/0.3% routers and network ≈2 kbps.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:11A new approach to reporting failures in distributed systems
- 2:23This design choice is mismatched to today's requirements
- 3:02Applications choose a recovery action that must work for all failures
- 5:00Strawman: expose individual failure type to application
- 5:53Our approach: group remote failures that applications handle similarly
- 9:26Implementing the interface is easy but...
- 11:15Distinguishing host and network failures reduces unavailability by 4x
- 13:26Detecting failures quickly helps Cassandra optimize replica selection
- 15:16Distinguishing transient and long-lasting failures helps avoid unnecessary recovery For a transient routing problem
- 16:06The interface is generally applicable
- 17:10Other related work: network monitoring
- 17:56Take-away points
related talks
16mPilot Execution: Simulating Failure Recovery In Situ for Production Distributed Systems
57mReliability Nirvana
15mFiDe: Reliable and Fast Crash Failure Detection to Boost Datacenter Coordination
22mFailing Over without Falling Over
45m⚡A Resilient Application in a Partially Degraded World ⚡
34mDistributed Systems - What Can Go Wrong and How to Get It Right
29mHorrors of Distributed Systems
01h 12mRedundancy does not imply fault tolerance