Reliability @Scale 2023
2023-09-25..2023-09-26
-- online --
[Reliability @Scale Summer 2022, Reliability @Scale 2023, Reliability @Scale 2024]
as the organisers put it
Join Reliability @Scale, a conference for passionate engineers building and understanding highly resilient systems at massive scale. Register today.
who it's for
site reliability engineers, reliability engineers, platform engineers, observability engineers, engineering managers, incident responders, data engineers
topics
liquid cooling for AI racks; heterogeneous GPU rollouts; ICSP paced rollouts; site degradation disaster recovery; zero-notice power storm testing; fine-tuning LLMs for incidents; time series monitoring store; blackbox alert workflow testing; redis to cassandra + S3 migration; client-side availability processor; advertiser value lost metric; reliability maturity model
| title | author | length |
|---|---|---|
| Reliability A to Z | Francois Richard, Surupa Biswas, Chris Malone, Syamla Bandla, Jason Kalich | 34m |
Meta describes building a new data center design with liquid cooling to support AI racks that are >4x denser, plus a pivot to heterogeneous GPU generations and… | ||
| Evolution of Disaster Recovery @ Meta | Ahmed Eid, Raghavendra Prabhu | 22m |
Meta documents a shift from a fixed extra-capacity DR buffer to a scalable site degradation strategy that deliberately sheds noncritical work to survive… | ||
| Large Language Models for Automatic Cloud Incident Management | Rujia Wang | 19m |
A system predicts root cause and mitigation at incident creation by fine-tuning foundation LLMs and evaluating against production data; fine-tuned DaVinci-002… | ||
| Building Resilient Monitoring at Meta | Adam Phillabaum, David Pariag | 18m |
Meta hardens monitoring that watches millions of hosts by building a Time Series store, petabytes of logs, tracing, and an end-to-end blackbox tester called… | ||
| Learnings & Patterns from Zero-Downtime Migrations | Kishore Banala, Anoop Panicker | 19m |
Conductor migrates its primary persistence from Redis-based Dynamite to Cassandra + AWS S3, moves queues from Dyno queues into a new Timestone system, and uses… | ||
| Scribe: Improving Reliability One 9 at a Time | Mohamed Bassem, Tiziano Carotti, Yuri Dolgov | 22m |
Scribe raises read availability from ~four nines to five nines by adding concrete measures: a client-side availability processor that emits three states… | ||
| Improving Reliability by Data-Driven Engineering & Culture Changes | David Amsallem, Gursharan Singh | 22m |
Defines a longitudinal metric called advertiser value lost and decomposes it into instantaneous burn rate, detection latency, and mitigation time to prioritize… | ||
| Cultural Foundation of Operational Excellence: Amazon’s Mechanisms for Resiliency | Peter M. O'Donnell | 19m |
Claims that Amazon’s operational resilience rests on a repeatable culture and concrete mechanisms: 14 leadership principles drive norms that eliminate… | ||
| Building a Culture of Reliability | Casey Bouskill | 20m |
Conducts over 40 interviews with engineers worldwide, codes responses in MAXQDA, and runs three survey waves with cognitive testing to track reliability… | ||
| A Reliability Maturity Model Explained | Tracy Ferrell, Vartika Agarwal | 22m |
A five‑level reliability maturity model (absent, reactive, proactive, strategic, visionary) maps seven attributes—operational processes, risk management… | ||
| Live Q&A with Speakers – Laine Campbell | Laine Campbell | 24m |
Live Q&A digs into concrete tradeoffs and fixes: measured cost of site degradation/MRDR by SLAs and user engagement; anti-entropy gaps in Dynamite → Cassandra… | ||
| Live Q&A with Speakers – Marina Fisher | Marina Fisher | 27m |
Panelists argue for an operational playbook: make asking “why” standard, run blameless postmortems, shrink blast radius and celebrate near misses, and treat… | ||
| Closing Remarks | Francois Richard | 01m |
Wraps the conference and confirms additional fall events focused on mobile system and video, lists distribution channels on Facebook, YouTube, and LinkedIn… | ||
| nothing matches | ||