SREcon24 Americas
2024-03-18..2024-03-20
San Francisco, United States
who it's for
site reliability engineers, platform engineers, observability engineers, engineering managers, cloud architects, security engineers, frontend engineers
topics
observability 2.0 single event store; columnar telemetry storage (ClickHouse); OIDC for CI/CD credential rotation; high cardinality metric mitigation; sharding strategies and tooling; capacity reservations and autoscaling; synthetic monitoring design and orchestration; incident severity and triage models; cache inconsistency in distributed systems; migrating large search datasets; compliance-friendly continuous delivery; frontend reliability gaps
| title | author | length |
|---|---|---|
| 20 Years of SRE: Highs and Lows | Niall Murphy | 27m |
Starts from a concrete origin: production was managed by a forked cluster.py list, teams built babysitter to edit arrays at runtime, then evolved into Borg and… | ||
| Scam or Savings? A Cloud vs. On-Prem Economic Slapfight | Corey Quinn | 28m |
Argues that cloud is usually chosen for speed and features, not lower cost, and that moving back on‑prem is rare except for small proofs of concept or… | ||
| Is It Already Time To Version Observability? (Signs Point To Yes.) | Charity Majors | 39m |
Argues observability needs a major-version bump: Observability 2.0 replaces fragmented metrics+logs+traces with a single source of truth of wide structured… | ||
| Capacity Constraints Unveiled: Navigating Cloud Scaling Realities | Kevin Sonney, Marc-Andre Dufresne | 29m |
Elastic Cloud runs across 64 regions on four CSPs with roughly 40,000 hosts supporting about 50,000 deployments, and capacity shortfalls happen despite… | ||
| Sharding: Growing Systems from Node-scale to Planet-scale | Adam Mckaig | 37m |
Stripe runs a multi-petabyte cluster and the concrete advice is: extract a stable shard key, map it to shards, then iterate — start with read replicas (often… | ||
| Product Reliability for Google Maps | Micah Lerner, Joe Abrams | 25m |
Maps reliability shifts from server SLOs to direct user signals for a billion monthly users, run by a ~20‑person SRE team, claiming it catches issues server… | ||
| Build vs. Buy in the Midst of Armageddon | Reggie Davis | 32m |
A 10-person SRE team collapses to 3, rebuilds an in-house incident tool Riot (Go), and discovers rapid commit churn, hidden maintenance costs and a… | ||
| Compliance & Regulatory Standards Are NOT Incompatible with Modern Development.. | Charity Majors | 37m |
Argues that no law forbids practices like continuous deployment, observability-driven development, feature flags or testing in production and shows real-world… | ||
| The Ticking Time Bomb of Observability Expectations | David Caudill | 38m |
Argues that “monitor everything” plus vendor-driven marketing creates runaway cost and noise: observability often consumes 20% of cloud bill, floods teams with… | ||
| Synthesizing Sanity with, and in Spite of, Synthetic Monitoring | Daniel O'Dea | 44m |
Four years of in-house work yields a concrete stack: Pollinator (browser runners) and PM2 for orchestration, used by ~30–40 teams across Jira. Tests run on a… | ||
| Migrating a Large Scale Search Dataset in Production in a Highly Available... | Leila Vayghan | 20m |
Shopify migrates Elasticsearch data across GCP regions while running clusters on GKE, using CFA for the indexing pipeline and SQL as the primary store. The… | ||
| OIDC and CICD: Why Your CI Pipeline Is Your Greatest Security Threat | Mark P Hahn, Ted Hahn | 17m |
Claim: CI pipelines hold long-lived credentials that leak and become the largest attack surface, so replace static tokens with OIDC tokens and tie them to… | ||
| When Your Open Source Turns to the Dark Side | Dotan Horovits | 34m |
Relicensing and sabotage are real risks: Terraform → BSL and HashiCorp’s fork ecosystem lead to OpenTofu (Linux Foundation) uptake; Elasticsearch → SSPL →… | ||
| The Sins of High Cardinality | Jef Spaleta | 20m |
Prometheus metrics with free‑form labels like source/destination IPs and trace IDs cause explosive series growth — simple examples jump from 125 to 2,500 lines… | ||
| Optimizing Resilience and Availability by Migrating from JupyterHub to the... | David Hoover, Alexander Perlman | 19m |
Migrating from JupyterHub to the Kubeflow Notebook Controller replaces a single-replica Hub API and in-memory CHP routing with a Kubernetes-native operator… | ||
| 99.99% of Your Traces Are (Probably) Trash | Paige Cruz | 35m |
Sampling reduces trace volume by design: use OpenTelemetry SDKs and the OpenTelemetry collector with head (probabilistic, rate-limit, adaptive) and tail… | ||
| Meeting the Challenge of Burnout | Christina Maslach | 38m |
Burnout is an occupational phenomenon from chronic workplace stressors, not a personal illness, and WHO’s 2019 review of 40 years of research frames it as a… | ||
| What We Want Is 90% the Same: Using Your Relationship with Security for Fun.. | Lea Kissner | 31m |
Claim: SRE and security want about 90% of the same outcomes and can combine levers to avoid surprises. Evidence: random failures scale with size (law of large… | ||
| Thawing the Great Code Slush | Maude Lemaire | 38m |
After October 2020 outages a live, Zoomed change advisory board (CAB) freezes Chef repo for seven months and costs roughly $5,000 weekly; response is to build… | ||
| Resilience in Action | Will Gallego | 36m |
Resilience is framed as adapting to the unexpected and building adaptive capacity — not perfect runbooks but the network, rituals and experiments that let… | ||
| Navigating the Kubernetes Odyssey: Lessons from Early Adoption and Sustained... | Raul Benencia | 35m |
Journey from scavenged servers to global Kubernetes infrastructure shows concrete choices and outcomes: bootstrapped K8s v1.12 on CoreOS, replaced with Ubuntu… | ||
| "Logs Told Us It Was Kernel – It Wasn't" | Valery Sigalov | 30m |
Investigation shows a 30% slowdown on a tiny loop and a clock_gettime jump from 17ns to 47ns that falls back to 23ns after boot-parameter fixes; root causes… | ||
| What Is Incident Severity, but a Lie Agreed Upon? | Em Ruppe | 35m |
Argues severity is a negotiated artifact, not a truth: a spreadsheet used a percentage of events unavailable within 24 hours threshold that teams could not… | ||
| Hard Choices, Tight Timelines: A Closer Look at Skip-level Tradeoff Decisions... | Laura Maguire, Courtney Nash | 49m |
Analysis mines a public corpus of over 10,000 incident reports (the the VOID), then runs a vignette study with 27 participants using scenario probes and… | ||
| Triage with Mental Models | Marianne Bellotti | 37m |
Explains why experts triage by matching observations to a large library of small, testable mental models and gives concrete exercises: build process maps to… | ||
| Defence at the Boundary of Acceptable Performance | Andrew Hatch | 31m |
Applies Jens Rasmussen’s Dynamic Safety Model to tech companies and claims the economic boundary is the strongest and the performance boundary the weakest… | ||
| Lightning Talks | Laura Nolan, Chitkala Sethuraman, Prathamesh Sonpatki, J. Paul Reed, Ted Hahn, Cail Young, Tushar Gupta, Adam Mckaig, Corey Quinn | 39m |
Nine micro-updates: cognition research shows experts use recognition and mental simulation with limits of three moving parts and ~six steps so planning lets… | ||
| System Performance and Queuing Theory - Concepts and Application | Jeff Poole | 40m |
Queueing theory explains why systems with 80% utilization still see long delays: variability in arrivals or service makes queues form and average residence… | ||
| It Is OK to Be Metastable | Aleksey Charapko | 38m |
Defines a metastable failure as a positive feedback loop that keeps a system degraded after a trigger and argues avoiding the metastable vulnerable state is… | ||
| The Art of SRE: Building People Networks to Amplify Impact | Daria Barteneva | 39m |
Argues that structured cross-team practice groups replicate choir techniques to amplify learning: sets up weekly 20–60 minute retrospective review “choirs”… | ||
| Teaching SRE | Mikey Dickerson | 46m |
Designed course "Managing Complex Systems" at Pomona College that turns CS majors into SREs by having durable teams of 4–5 build and run a messy, real web app… | ||
| Cross-System Interaction Failures: Don't Fail through the Cracks | Tianyin Xu, Xudong Sun | 40m |
Empirical study of 120 cross-system incidents and a random sample of 360 issues finds roughly 30% are CSI failures that arise only when systems interact… | ||
| Gray Failure: The Achilles’ Heel of Cloud-Scale Systems | Ze Li, Ryan Huang | 44m |
Argues that subtle “gray failures” arise when built-in detectors disagree with application symptoms, calls this differential observability, and shows a… | ||
| The Invisible Door: Reliability Gaps in the Front End | Isobel Redelmeier | 43m |
Front end reliability focuses on crashes, TTI, jank and ANRs, not just CPU, and often requires high‑cardinality metrics and heavy sampling. A pilot team… | ||
| Automating Disaster Recovery: The Ultimate Reliability Challenge | Ricard Bejarano | 25m |
Claims day Zero reliability is required and shows a pragmatic patch: Cisco automates cross‑region recovery for 215 services by shipping a standby data copy… | ||
| From Chaos to Clarity: Deciphering Cache Inconsistencies in a Distributed... | Akashdeep Goel, Prudhviraj Karumanchi | 15m |
A distributed EVCache deployment spans 194 clusters and 22,000 servers storing 2 trillion items (14.3 PB) and its replication engine handles 30 million… | ||
| Patching Your Way to Compliance with a Small Team and a Pile of Technical Debt | Filipe Felisbino | 18m |
A three‑pronged strategy cuts a mountain of patches down to manageable work: pick core technologies, decommission low‑value systems, and move remaining… | ||
| Strengthening Apache Pinot's Query Processing Engine with Adaptive Server... | Jia Guo | 40m |
Apache Pinot at LinkedIn adds Adaptive server selection and Runtime query killing to handle 1,500+ brokers, 8,000+ servers, 4,500 tables and ~300K QPS… | ||
| Taming the Linux Distribution Sprawl: A Journey to Standardization and... | Raj Shekhar, James Fotherby | 16m |
A company narrows dozens of distributions to two hardened images — Amazon Linux 2 and CentOS 7.9 — selected because they are RPM‑based, prioritize stability… | ||
| Frontend Design in SRE | Andreas F. Bobak | 17m |
Claims that SRE tooling must shift from training-heavy pet tools to scalable cattle-facing frontends that non-SREs can use; recommends three concrete… | ||
| Measuring Reliability Culture to Optimize Tradeoffs: Perspectives from an... | Kathryn (Casey) Bouskill | 18m |
Conducted over 40 interviews and four survey waves, the team uses MAXQDA to code themes and finds 78% say teams value reliability while about half still can’t… | ||
| Storytelling as an Incident Management Skill | Laura de Vesine | 19m |
Argues that incident narratives must show the logical progression of cause and effect not just who did what and when Tools: architecture diagrams, telemetry… | ||
| Real Talk: What We Think We Know — That Just Ain’t So | John Allspaw | 37m |
Argues that several SRE assumptions fail when tested: the lines of code metric is discredited (Capers Jones, 1995); a widely quoted 85% change claim (Gartner… | ||
| What Can You See from Here? | Tanya Reilly | 49m |
Argument: your immediate context warps priorities and fuels pointless fights; concrete example: bike-shedding origin is a 1999 FreeBSD thread about sleep… | ||
| nothing matches | ||