Reliability @Scale 2024
2024-10-09
-- online --
[Reliability @Scale Summer 2022, Reliability @Scale 2023, Reliability @Scale 2024]
who it's for
production engineers, site reliability engineers, platform engineers, release engineers, observability engineers, engineering managers, database engineers
topics
GPT-4 incident diagnosis; canary phase rollout; Retriever columnar datastore; safe deployment methodologies; nine SLIs taxonomy; regional config validation; zero-downtime schema migrations; WWW release automation; dependency safety patterns; deterministic replay testing; NVMe plus S3 tiering; unified signals pipeline
| title | author | length |
|---|---|---|
| Opening Remarks | Michael Chang | 02m |
A recent partial outage causes a significant portion of app traffic to drop, a bad deploy is reverted within minutes, and teams rebuild the rollout process… | ||
| Closing Remarks | Michael Chang | 01m |
Change is constant and moving through changes safely is presented as the metric that keeps a business running, with a concrete commitment to evolve techniques… | ||
| Improving Cloud Reliability at Scale Using Gen AI | Chetan Bansal | 14m |
Describes a deployed framework that uses GPT‑4–based agents to automate incident diagnosis by integrating telemetry, runbooks and debugging tools into a… | ||
| Safe Deployment: Exploring Methodologies | Francisco Bautista, Abhinav Sharma | 18m |
Meta describes a Safe Deployment program that moves from unit and integration tests to a canary phase that can qualify changes by the next day, then does slow… | ||
| Building & Running a Scuba-Inspired Observability Tool | Liz Fong Jones, Ben Hartshorne | 23m |
Honeycomb builds Retriever, a columnar datastore that pairs NVMe-local shards and Kafka ingest with S3 tiering and tens of thousands of concurrent Lambdas for… | ||
| Reliability, Innovation, Time to Market: Choose All Three | Justin Gibbs | 19m |
Claims that reliability, innovation and speed are simultaneously achievable by making reliability a hard requirement and running parallel certification… | ||
| 9 SLIs; Oh My! | Sal Furino | 20m |
Proposes nine concrete SLIs grouped into three sets: customer/user experience, supporting services, and management/reputation. Gives concrete examples and… | ||
| A Decade of Progress: The Evolution of Production Engineering and Reliability | Francois Richard, Pedro Canahuati, Peter Hoose | 55m |
History of Production Engineering at Meta: how appops and SRE split into production engineering and site reliable operations, concrete automation wins (mcash… | ||
| Limiting the Blast Radius: Preventing Outages from Bad Configs with Datacenter-Scale Health Checks | Joseph Romano, Zach Zundel | 18m |
Meta deploys a datacenter-scale canary called regional config validation that tests a config in one canary region for about 10 minutes and blocks global… | ||
| Scaling Releases: Inside Meta's WWW Release Operations | Vladimirs Kotovs, Casey McGinty | 21m |
Meta runs continuous www releases with 10 releases per day, an average 4–5 hour change release time, and automation handling 95% of releases without operator… | ||
| Making Deployments Safe at Meta | Anton Korenkov, Christopher Hegre | 18m |
Meta runs tens of thousands of changes every day for over 3.2 billion users and now gates rollouts with phased regional pushes, staging, and a set of default… | ||
| Applying Continuous Delivery to Database Schema | Tyson Trautmann | 21m |
Claim: applying Continuous Delivery to database schema cuts deployment risk and speeds releases by putting schema changes through automated pipelines. Uses… | ||
| Dependency Safety for Distributed Systems | Ankita Vimal, Antonio De La Vega | 21m |
Meta frames dependency risk as the fourth leading cause of outages and reports sub-50% test coverage on runtime paths, so it builds a policy to remove cycles… | ||
| Resilience Is Not Enough | Joe Magerramov | 22m |
Argues that redundancy alone fails and systems must be built to recover quickly and predictably, not just tolerate faults. Recommends concrete practices: fault… | ||
| Live Q&A Session #3 | Joe Gasperetti | 24m |
Roundtable covers concrete practices: adopt Honeycomb-style observability with a three-tier oroboros deployment that uses dynamic sampling to avoid feedback… | ||
| Live Q&A Session #2 | Joe Gasperetti | 25m |
A live panel explains concrete reliability practices: a regional config validation system (RCV) that uses real user traffic and canaries to catch site-scale… | ||
| Live Q&A Session #1 | Joe Gasperetti | 21m |
A live panel walks through concrete practices: SLO tooling gets tailored to decision personas with dashboards that show uptime versus error windows and… | ||
| nothing matches | ||