Understanding, Detecting and Localizing Partial Failures in Large System Software
summary
OmegaGen automatically generates customized, stateful watchdogs by using program reduction to build mimic checkers that exercise the same operations as the main module while isolating replicated context. A study of 100 real partial failures across five systems motivates the approach; OmegaGen is applied to six distributed systems, reproduces 22 real cases, detects 20 with median detection 4.2 seconds and localizes 18, finds an unknown ZooKeeper bug, and incurs ~1% false alarms and 5–6.6% throughput impact.
AI-written summary · about
chapters (32)
- 0:00Intro
- 1:04The failure off the radar
- 1:32Modern software is complex
- 2:15Study methodology
- 2:58root causes are diverse a There is no single uniformed or dominating root cause
- 3:20nearly half cases cause stuck issues
- 3:5115% cases are totally silent
- 4:17most cases are triggered by unique production workload or environment
- 4:54debugging time is long
- 5:25How to deal with partial failures
- 5:50Manual vs generated checkers
- 6:34Design principle: checkers intersect with the execution of a monitored process
- 6:47Intrinsic watchdog: Runtime
- 7:47Core idea: mimic checking
- 8:17Tool overview
- 8:41What is program reduction?
- 9:192 We need not put everything into checker
- 10:08locate long-running regions
- 10:33reduce the program
- 10:45locate vulnerable operations
- 11:02encapsulate watchdog checkers
- 11:13insert watchdog hooks
- 11:27Validation
- 11:49Prevent Side Effects
- 12:58Evaluated systems
- 13:1622 real-world partial failures reproduced for evaluation
- 13:45Detection Time
- 14:22Detection Localization
- 15:01Discover new bug
- 15:45False alarms
- 16:16System overhead
- 16:45Conclusions
related talks
40mAutomatic Generation of Runtime Checkers for Production Distributed Systems
22mFailing Over without Falling Over
16mEngineering Failure-Resilient Systems
44mGray Failure: The Achilles’ Heel of Cloud-Scale Systems
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
16mHolmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters
51mImproving resiliency with correction of error processes
34mWhy Good Software Goes Bad