Scaling Reliability: So You Want to Add a 9
summary
Goal is move from brittle multi‑service stacks toward concrete SLAs like 99.9% success by using actual tools: Finagle provides connection pooling, graceful goaway drains, retries and circuit breakers; use linkerd/sidecar for non‑JVM services and DTabs for staging routing. Detect failures with EWMA and trace internal retries, enforce a retry budget (example: cap retries at 25%), use canaries and rollbacks, and apply backpressure/rate‑shed to stop retry storms — these changes measurably raise end‑to‑end success (e.g. to ~99.6%).
AI-written summary · about
chapters (38)
- 0:00Introduction
- 0:15Reliability
- 0:43Who am I
- 1:55Who Cares About Reliability
- 2:39How Do We Measure Reliability
- 4:15Own Your Own Failures
- 4:44Background Noise Failures
- 5:43probabilistic failure
- 6:42impact reach high
- 7:15bad day
- 8:01failure detection
- 9:00ride around
- 9:17load balancing
- 9:31error codes
- 10:20microservice architecture
- 10:45overall success rate
- 11:13failures on edges
- 12:37the caller
- 12:56the service
- 14:49latency
- 15:19observability
- 16:32rollback
- 17:17rollback example
- 17:33rolling failures
- 19:19more problems
- 19:47planned events
- 22:15retry storms
- 22:31retry budget
- 23:01cluster overload
- 24:28request not getting response
- 25:23bad neighbors
- 27:56toggles
- 28:36staging
- 30:06other teams
- 32:03failing collaboratively
- 34:54finagle
- 43:01Is Fif extensible
- 44:36How would you participate in an ecosystem
related talks
41mSLOs that Delight: Master Multi-Service SLO Calculations
41mScaling Reliability Engineering with Tools
51mDesigning for Failure: Build Resilient Services
38mImplementing SRE practices: SLI/SLO deep dive
35mSCALA BASED SERVICES RESILIENCE IN THE REAL WORLD
15mShit, our Site is Down! Let's build a culture of reliability
01h 00mBuilding Reliable Services at NRK TV
27mContinuous Reliability. How?