SREcon21
as the organisers put it
SREcon21, a gathering of engineers who care deeply about site reliability, systems engineering, and working with complex distributed systems at scale, will be held as a virtual event for the global SREcon community on October 12–14, 2021.
| title | author | length |
|---|---|---|
| When Linux Memory Accounting Goes Wrong | Minhaj Ahammed | 27m |
LinkedIn searcher nodes hit host OOMs because cgroup page-cache accounting fails when a container is restarted and its cgroup is deleted and recreated: shared… | ||
| Don't Follow Leaders or "All Models Are Wrong (and So Am I)" | Niall Murphy | 45m |
Five years on, the SRE book is credited with spreading ideas but also fossilizing practice: widespread copying produces stalled innovation. Claims: SLOs and… | ||
| Let the Chaos Begin—SRE Chaos Engineering Meets Cybersecurity | Francesco Sbaraglia, Adriana Petrich | 25m |
Shows a concrete Security Chaos Engineering run against a Kubernetes cluster that deploys a kubernetes honeypot and measures detection with a Prometheus… | ||
| What's the Cost of a Millisecond? | Avishai Ish Shalom | 30m |
Claim: tiny delays amplify everywhere because queues, head‑of‑line blocking and retries compound multiplicatively. Uses an M/M/1 simulation, Kingman’s formula… | ||
| What To Do When SRE is Just a New Job Title? | Benjamin Bütikofer | 13m |
Company migrates from datacenters to cloud for a platform with 4M users and ~5M items/year, leaving ops people without purpose; solution is to define a clear… | ||
| Capacity Management for Fun & Profit | Aly Fulton | 30m |
Elastic Cloud runs on AWS GCP and Azure with stateful VM allocators sold in RAM increments and a 58 GB block size, and capacity work moves from manual PR’d… | ||
| A Political Scientist's View on Site Reliability | Michael Krax | 30m |
Argues that team change in knowledge work only succeeds by small experiments led by motivated members, not top-down edicts, because hierarchical power is… | ||
| Panel: Engineering Onboarding | Jennifer Petoff, Anne Hamilton, Sandi Friend, Ilse White, Daria Barteneva | 50m |
Conference panel lays out concrete numbers and fixes: research finds 83% vs 13% satisfaction (organizations think onboarding works but new hires disagree)… | ||
| Sparking Joy for Engineers with Observability | Zac Delagrange | 13m |
A production bug takes five days to fix before integration of exception reporting with distributed tracing; after adding a conversation header from the browser… | ||
| Panel: Observability | Liz Fong Jones, Gabe Wishnie, Štěpán Davidovič, Richard Waid, Partha Kanuparthy, Daria Barteneva | 50m |
Panel argues observability requires concrete tradeoffs not slogans, showing tools and limits: OpenTelemetry focuses on vendor‑neutral SDKs but not backend… | ||
| 10 Lessons Learned in 10 Years of SRE | Andrea Spadaccini | 29m |
Ten years of SRE experience across Google and Azure yields concrete rules: write down goals and measure progress toward stated goals; staff with a mix of… | ||
| Rethinking the SDLC | Emily Freeman | 39m |
Argues the sixty-year SDLC is obsolete and proposes the Revolution model that maps five concentric roles—architecting (largest), automating development… | ||
| Elephant in the Blameless War Room—Accountability | Christina Tan, Emily Arnott | 28m |
Shows that blamelessness and accountability coexist: hold people only after exhausting systemic fixes and clear prerequisites (were expectations documented… | ||
| How LinkedIn Performs Maintenances at Scale | Akash Vacher | 20m |
LinkedIn runs hundreds of thousands of servers across global data centers and uses impact analysis to map affected hosts and services for a maintenance… | ||
| Take Me Down to the Paradise City Where the Metric Is Green and Traces Are Pretty | Ricardo Ferreira | 26m |
Practical walkthrough of OpenTelemetry using Elastic Observability: deploy an OpenTelemetry collector in Docker Compose, point it at an Elastic Cloud… | ||
| Need for SPEED: Site Performance Efficiency, Evaluation and Decision | Kingsum Chow, Zhihao Chang | 27m |
Shows that common CPU utilization numbers lie: an Intel two‑socket 16‑core system with hyperthreading yields 64 logical CPUs, and running SPECjbb2005 pinned to… | ||
| SLX: An Extended SLO Framework to Expedite Incident Recovery | Qian Ding, Xuan Zhang, Xuan Zhang | 31m |
SLX extends SLOs to manage hundreds of objectives for a 99.999% availability target across 60+ Kubernetes clusters and 50+ components, building a dependency… | ||
| Watching the Watchers: Generating Absent Alerts for Prometheus | Nick Spain | 13m |
Prometheus rules parser extracts every selector from alert and recording rules, auto-generates a matching absent alerts rule by wrapping selectors in… | ||
| A Principled Approach to Monitoring Streaming Data Infrastructure at Scale | Eric Schow, Praveen Yedidi | 12m |
CrowdStrike ingests over a trillion events per day and models monitoring along two axes to produce four quadrants: observability, operability, availability… | ||
| Let's Bring System Dynamics Back to CS! | Marianne Bellotti | 27m |
Argues that outages come from feedback loops and coupling, not impossible states, and uses the AWS Kinesis 2019 outage as a concrete example of memory/map… | ||
| From 15,000 Database Connections to under 100—A Tech Debt Tale | Sunny Beatteay | 30m |
Company reduces MySQL connections from 15,000 database connections to under 100 by replacing per-hypervisor polling with a regional event router, rewriting the… | ||
| MySQL and InnoDB Performance for the Rest of Us | Shaun O'Keefe | 34m |
Practical checklist for fixing slow MySQL/InnoDB queries: use EXPLAIN and SHOW FULL PROCESSLIST to see planner choices and thread states; prefer small integer… | ||
| Cache Strategies with Best Practices | Tao Cai | 26m |
Describes concrete tactics: introduce soft and hard TTL plus async refresh so expired keys keep serving while background refreshes run, and dynamic TTL based… | ||
| Optimizing Cost and Performance with arm64 | Liz Fong Jones | 31m |
Honeycomb.io evaluates and then migrates core services to arm64 over a year, moving 92% of compute workloads to Graviton2 m6g instances; ingestion service… | ||
| Ceci N'est Pas un CPU Load | Thomas Depierre, John Allspaw, Paul Hammond | 16m |
Claim: digital systems lose the clean, probeable input→output plumbing of analog electronics, so common observability tools mislead. Shows a relay punch‑card… | ||
| Grand National 2021: Managing Extreme Online Demand at William Hill | Matthew Berridge, Josh Allenby | 15m |
William Hill precomputes a 16x peak target by splitting users into cohorts, then tests, tunes, and scales services, offloading to cloud and caching. Traffic… | ||
| Microservices above the Cloud—Designing the International Space Station for Reliability | Robert Barron | 29m |
The International Space Station is built from ~50 modules added over decades, giving modular upgrades, redundant subsystems and surprising edge cases: three… | ||
| Horizontal Data Freshness Monitoring in Complex Pipelines | Alexey Skorikov | 22m |
Google builds a curated data dependency graph from harvested access logs, metadata (mod time, writer, TTL, logical size) and distributed traces, then runs… | ||
| How We Built Out Our SRE Department to Support over 100 Million Users for the World's 3rd | Sinéad O'Reilly | 31m |
Aspiegel scales its SRE department from ~25 to over 100 engineers during 2020 to support Huawei’s mobile ecosystem (millions of users, 60 countries, 100+… | ||
| You've Lost That Process Feeling: Some Lessons from Resilience Engineering | David D. Woods, Laura Nolan | 30m |
Argues that process feel—a shared, dynamic sense of where a system ought to be—is essential to handle anomalies, and that designing displays which show “where… | ||
| Scaling for a Pandemic: How We Keep Ahead of Demand for Google Meet during COVID-19 | Samantha Schaevitz | 13m |
February–March 2020 incident response organizes 100+ engineers across 10 teams into parallel work streams (capacity, bottlenecks, dependencies, production… | ||
| DevOps Ten Years After: Review of a Failure with John Allspaw and Paul Hammond | John Allspaw, Thomas Depierre, Paul Hammond | 49m |
Ten years on, the claim that teams ran continuous deployment and even developers deploy their own code is defended with concrete shifts: configuration… | ||
| What If the Promise of AIOps Was True? | Niall Murphy | 34m |
Defines AIOps as ML applied to production infrastructure, contrasts three stances: impossible today because production lacks the sample sizes and ROI, an… | ||
| Model Monitoring: Detecting and Analyzing Data Issues | Dmitri Melikyan | 12m |
Models fail silently when upstream data changes or becomes all zeros, so production needs monitoring that detects anomalies and drift rather than just uptime… | ||
| Leveraging ML to Detect Application HotSpots [@scale, of Course!] | Sanket Patel | 14m |
LinkedIn builds a daily pipeline that trains per-service regression models in Python (caption notes a "salary framework") using 15 days of training data… | ||
| Demystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform | Mary McGlohon | 29m |
Describes a production ML stack: feature-generation pipelines, a batch model-training service with orchestration, an inference service and an online feedback… | ||
| Designing an Autonomous Workbench for Data Science on AWS | Dipen Chawla | 13m |
Built a self-service ML workbench using JupyterHub on AWS EKS with KubeSpawner to provision notebook servers, EFS persistent volumes per user, and node pools… | ||
| Panel: OpML | Todd Underwood, Josh Hartman, Zhangwei Xu, Nisha Talagala, Vanessa Yiu | 46m |
Operationalizing ML faces subtle failures and distribution shifts that are hard to detect — a schema drift takes three weeks to diagnose in one recommendation… | ||
| When Systems Flatline—Enhancing Incident Response with Learnings from the Medical Field | Sarah Butt | 15m |
Maps medical protocols to incident work: applies ACLS-style algorithm-guided decisions for fast, standardized triage, borrows ATLS principles to stop the… | ||
| Evolution of Incident Management at Slack | D. Brent Chapman | 28m |
Slack handles over 150 million messages per minute and now runs about a dozen controlled deployments per day using dog food and canary phases with phased… | ||
| Hacking ML into Your Organization | Cathy Chen | 15m |
A case study of Yarnet.ai shows a proof of concept that boosts sales and then breaks because organizational gaps — managers don’t know what normal model… | ||
| Automating Performance Tuning with Machine Learning | Stefano Doni | 22m |
Claims an automated reinforcement learning optimizer applies configurations via Kubernetes APIs runs Locust load tests and collects Prometheus and Istio… | ||
| Practical TLS Advice for Large Infrastructure | Mark P Hahn, Ted Hahn | 31m |
Argues that TLS must be applied everywhere in large deployments and that ACME or service meshes alone do not solve certificate authority architecture… | ||
| Learning More from Complex Systems | Andrew Hatch | 28m |
Four concrete takeaways: incidents are emergent novel events that teach more than root‑cause hunting; move from reductionist “one cause” analysis to… | ||
| Of Mice & Elephants | Koon Seng Lim, Sandeep Hooda | 30m |
A production job writes thousands-deep nested paths and triggers constant failures that, combined with an object-store firmware bug, causes the cluster to be… | ||
| User Uptime in Practice | Anika Mukherji | 26m |
Pinterest implements User Uptime (from Google's Meaningful Availability) as a time-based metric by logging user events from a Python Flask API to a dedicated… | ||
| Nothing to Recommend It: An Interactive ML Outage Fable | Todd Underwood | 30m |
An online yarn shop uses ML to rank search and surface recommendations, retraining the same model daily retraining and deploying unchanged config. Customers… | ||
| Improving Observability in Your Observability: Simple Tips for SREs | Dan Shoop | 26m |
Argues that time‑series graphics date back centuries and extracts concrete practices: Playfair’s 43 of 44 charts are time series and Minard packs six variables… | ||
| SRE for ML: The First 10 Years and the Next 10 | Todd Underwood | 44m |
Built a production ML stack for ad selection since ~2009: data ingestion and preprocessing, training cluster, a model management and pipeline orchestration… | ||
| Lessons Learned Using the Operator Pattern to Build a Kubernetes Platform | Pavlos Ratis | 20m |
Lessons Learned Using the Operator Pattern to Build a Kubernetes Platform
Pavlos Ratis, Red Hat
In Kubernetes, the Operator pattern helps capture good… | ||
| Nine Questions to Build Great Infrastructure Automation Pipelines | Rob Hirschfeld | 27m |
Nine practical questions that reveal why CICD models fail for infrastructure and how to build predictable, reusable pipelines that move persistent resources… | ||
| Hard Problems We Handle in Incidents but Aren't Recognized | John Allspaw | 31m |
Identifies four concrete incident activities—diagnostic work, therapeutic actions, recruiting people, and status reporting—and shows how familiar tools… | ||
| Experiments for SRE | Debbie Ma | 28m |
Experiments use feature flags and staged ramps (example ramps: 1%, 10%, 50%) to limit blast radius, attribute outages to a treatment experiment id, and roll… | ||
| Reliable Data Processing with Minimal Toil | Pieter Coucke, Julia Lee | 30m |
Google and Slack implement automated global rollouts that promote batch-job versions through staged shards sliced by user id and traffic, with read-write… | ||
| SRE "Power Words"—the Lexicon of SRE as an Industry | Dave O'Connor | 15m |
SRE "Power Words"—the Lexicon of SRE as an Industry
Dave O'Connor, Elastic
As the SRE Industry develops, we've come to rely on certain words, phrases, and… | ||
| How Our SREs Safeguard Nanosecond Performance—at Scale—in an Environment Built to Fail | Jillian Hawker | 15m |
Optiver describes a low-latency, on-prem trading stack that moves 230 TB/day across ~1,500 servers to connect to 60+ exchanges using FPGAs and fiber/microwave… | ||
| Panel: Unsolved Problems in SRE | Kurt Andersen, Niall Murphy, Narayan Desai, Laura Nolan, Xiao Li, Sandhya Ramu | 45m |
Panel surveys where SRE practice actually breaks: contrasts consulting vs ownership model, warns one-size-fits-all SLOs, and argues for tool-driven… | ||
| Beyond Goldilocks Reliability | Narayan Desai | 39m |
Claims a calibration-free stationarity model that scores the relative likelihood of observed events, runs on historical telemetry and flags gradual, days-long… | ||
| A Retrospective: Five Years Later, Was Chaos Engineering Worth It? | Miko Pawlikowski, Sachin Kamboj | 30m |
Five years of practice yields concrete verdicts: chaos experiments are cheap, automatable regression checks that catch real bugs. Examples include kubelet… | ||
| The Origins of USAA's Postmortem of the Week | Adam Newman | 18m |
A company-wide weekly Postmortem of the Week meeting runs Thursdays at 11pm ET, gathers 200–400 attendees after a team’s own debrief, and focuses on polished… | ||
| Cache for Cash—Speeding Up Production with Kafka and MySQL binlog | Barak Luzon | 28m |
Company pushes MySQL changes into Kafka and MySQL binlog to update front-end caches instead of pulling, turning millions of background TTL refreshes into a… | ||
| Taking Control of Metrics Growth and Cardinality: Tips for Maximizing Your Observability | Rob Skillington | 27m |
Cloud-native metric volumes explode: a 10-endpoint precloud setup (~150,000 time series) can become 150 million time series when pods scale to 30,000 and… | ||
| Games We Play to Improve Incident Response Effectiveness | Austin King | 33m |
Surveys concrete exercises: party games like Jackbox/Drawful to build trust, communication drills such as Keep Talking and Nobody Explodes to practice calm… | ||
| Food for Thought: What Restaurants Can Teach Us about Reliability | Alex Hidalgo | 30m |
Compares restaurant failures — late plates, wrong or burnt food, incorrect bills — to SRE symptoms and maps concrete fixes: add staff to scale, hire runners… | ||
| Latency Distributions and Micro-Benchmarking to Identify and Characterize Kernel Hotspots | Danny Chen, Danny Chen | 26m |
Engineering uses micro-benchmarks and latency histograms to pinpoint kernel hotspots across AIX Linux and Solaris rather than rely on topic-level diagnostics… | ||
| Trustworthy Graceful Degradation: Fault Tolerance across Service Boundaries | Daniel Rodgers-Pryor | 23m |
Third of Australian students use the service, with tens of thousands online simultaneously, and a default 30 second timeout once makes the app feel totally… | ||
| Spike Detection in Alert Correlation | Nishant Singh | 20m |
LinkedIn builds a call graph of microservice dependencies and applies a modified zscore (median absolute deviation based) to each service metric time series to… | ||
| nothing matches | ||