Reliability @Scale Summer 2022
as the organisers put it
Reliability @Scale is a technical conference for engineers who are passionate about building and understanding highly resilient and reliable systems at massive scale. Whether it’s novel design decisions, or outages that impact billions of people, providing reliable experiences for Systems at this scale present unique technical challenges. The @Scale community focuses on bringing people together […]
who it's for
site reliability engineers, incident responders, infrastructure engineers, platform engineers, observability engineers, engineering managers
topics
backbone router drain recovery; DNS heartbeat failure; Consul streaming rollback; QUIC handshake null pointer; configuration rollout safety; overload protection and staged bring-backs; subsecond telemetry sampling; Auto Canary canary pipeline; SLO tracking with Sleek annotations; service degradation orchestration; UPS rectifier IGBT failure; missing shell option causing crash loop
| title | author | length |
|---|---|---|
| How We Drained Every Backbone Router Simultaneously | Santosh Janardhan | 12m |
A routine capacity command took all eight EBB planes offline and a simultaneous DNS heartbeat failure causes every name server to withdraw, effectively… | ||
| Lessons Learned from the Halloween Outage | Max Ross | 24m |
73-hour global outage at Roblox stems from enabling Consul streaming on load balancers which, 24 hours later, triggers a Consul performance collapse and then… | ||
| QUIC Exit: Exposing a New Class of Outage | Ian Swett | 15m |
A QUIC handshake null pointer in legacy code lets a Google front end issue poison encrypted tokens that clients replay to other GFEs, triggering a cascading… | ||
| Service Incident Deep Dive: Technical Overview & Learnings | James Kretchmar | 16m |
A July 2021 Akamai incident unassigns traffic for about an hour for a subset of customers after a semantically wrong change to a flexible assignment feature… | ||
| Lessons From Long-Running Investigations | Jana Iyengar, Hossein Lotfi | 21m |
Describes concrete investigative practices: using divide and conquer by running parallel experiments across a fleet to spot divergence and pinpoint a… | ||
| Infrastructure: Engineering for Resiliency at Scale | Prasad Kalyanaraman | 12m |
An unexpected December 16, 2021 event causes ~30 minutes of slowed customer requests when centralized UPS rectifiers fail to track a noisy utility, triggering… | ||
| PIPEFAIL: How a Missing Shell Option Slowed Cloudflare Down | Jeremy Hartman | 17m |
A bash cronjob running in Kubernetes silently returns an empty TOML when an addressing API query slows, because the script lacks strict error handling (missing… | ||
| Live Q&A Session #1 | Anca Agape | 42m |
Panel answers specific operational questions about outages and prevention: how automatic P0 paging fires when control‑plane/API or global traffic drops and… | ||
| Improving Reliability @ Meta: By Analyzing Historical Events That Led to SLO Violations | Kostiantyn Tsaregradskyi, Keshav Varma | 19m |
Instagram and Meta move SLO tracking from siloed dashboards and spreadsheets into a unified tool called Sleek, add a structured annotations schema with… | ||
| Service Degradation at Scale: Creating Instant Capacity | Thote Gowda, Yi Yu | 27m |
Service degradation disables noncritical features to reclaim compute instantly, with concrete targets of 5/10/20% CPU savings for Levels 3/2/1 and validated… | ||
| Shrinking the Impact of Production Incidents | Yuri Grinshteyn | 29m |
Argues that postmortems are the primary input for prioritizing reliability work and shows how to cut user pain by improving time to detect, time to mitigate… | ||
| Reliably Changing Configuration @ Scale | Avery Berchek | 21m |
Meta uses configurator and an automated Auto Canary pipeline that runs structured canary specs to test config changes on real workloads; tooling auto-generates… | ||
| SEV Culture: How Today's SEVs Create Tomorrow's Reliability | Joe Gasperetti, Nick Egebo | 18m |
Claim: filing more SEVs improves reliability, supported by cross‑industry studies showing airlines and construction sites with more incident reports have fewer… | ||
| Live Q&A Session #2 | Christian Monzon | 26m |
Live Q A summarizes concrete reliability practices and outcomes not platitudes it shows using Auto Canary with automated enrollments and exponential adoption… | ||
| nothing matches | ||