Scribe: Improving Reliability One 9 at a Time
summary
Scribe raises read availability from ~four nines to five nines by adding concrete measures: a client-side availability processor that emits three states (healthy, misbehaving, unknown) and latency computed by replacing timestamps at buffer entry and summing per-stage delays; dependency mapping from service-router Thrift traces plus networking logs; systematic dependency injection tests and weekly region-drain exercises (50+ runs) that surface tens of bugs; fixes reduce incidents like 98‑minute and 18‑minute read outages and prevent revenue loss.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:30Scribe Overview
- 4:15Key Themes
- 5:07Data Part
- 5:48Availability
- 7:34Availability Processor
- 8:35Previous Use Case
- 9:19Latency Monitoring
- 10:09Latency in Streaming
- 12:01Recap
- 13:11Dependencies
- 14:04Identifying dependencies
- 15:20Mitigation
- 16:09Testing
- 18:03Infra
- 19:22Capacity Loss
- 21:02Summary
related talks
42mThe History of Logging @ Facebook (Abridged)
14mSelective Reliability Engineering: There Is No Single Source...
35mRefining Systems Data without Losing Fidelity
26mMeasuring Success - Observability and More!
23mHow Atlassian is migrating 40PB data with 99.99% reliability
22mSiphon : Modern Data Stack with SF-CH & Iceberg
31mAdvancing security analytics with Snowflake and AWS at Snyk (PRT329)
01h 06mBest practices when accessing Big Data or any other data!