SREcon25 Americas
2025-03-25..2025-03-27
Santa Clara, United States
who it's for
site reliability engineers, production engineers, observability engineers, platform engineers, machine learning infrastructure engineers, incident responders, database reliability engineers
topics
OpenTelemetry distributed tracing; model monitoring and drift detection; GPU training cluster reliability; canary and progressive rollouts; slow SQL detection and prevention; log clustering with embeddings; chaos datacenter stress testing; OpenTelemetry semantic conventions; async pipeline observability and alerting; network flow and AWS networking limits; HTTP proxy performance tuning; multi-region and multi-cloud failover
| title | author | length |
|---|---|---|
| Safe Evaluation and Rollout of AI Models | Brendan Burns | 38m |
Azure Copilot for the Azure portal uses foundation models with retrieval augmented generation (no fine‑tuning) and ships through canary dogfooding and 1%… | ||
| Improving the SRE Experience for 10 Years as a Free, Open, and Automated... | Matthew McPherrin | 38m |
Let’s Encrypt issues free TLS for 500M+ websites and about 7M certificates/day, run by a 25‑person nonprofit (4 devs on Boulder, 9 on SRE) with a ~$4.5M annual… | ||
| An SRE Approach to Monitoring ML in Production | Daria Barteneva | 39m |
Claims that 91% of models degrade and that 50% of teams run no ML monitoring drive concrete practices: monitor four layers—infrastructure, service, data… | ||
| Transformers in SRE Land: Evolving to Manage AI Infrastructure | Qian Ding | 36m |
Ant Group describes managing thousand‑GPU training clusters where a single bad GPU causes daily multi‑hour interruptions and idle expensive hardware… | ||
| Tackling Slow Queries: A Practical Approach to Prevention and Correction | Kurni Famili, Brad Feehan | 32m |
Shopify reports peaks like $4.6M per minute and serves 10.5 trillion queries so slow SQL causes about one in five merchant‑impacting incidents. The build is… | ||
| The Search for Speed | Scott Laird | 29m |
A messy OpenSearch deployment on AWS becomes measurable: added probes, tracing spans and histograms, replaced OpenSearch Benchmark with a custom Go benchmark… | ||
| Case Study: A Thundering Herd in the Wild | Nicolas Arroyo | 35m |
Investigation traces extreme P95 outliers to thousands of unrelated threads waking simultaneously every 5 seconds due to recurring jobs aligned to the clock… | ||
| Techniques Netflix Uses to Weather Significant Demand Shifts | Joseph Lynch | 44m |
Netflix runs a four-region full-active control plane backed by Apache Cassandra, EVCache and Zuul and steers traffic via ~100 authoritative DNS points on Open… | ||
| Live, Laugh, Log | Paige Cruz | 40m |
Operational guide to Fluent Bit with concrete knobs, failure modes and diagnostics: how CRI log files hit the tail input, why the Kubernetes filter enriches… | ||
| Distributed Tracing in Action: Our Journey with OpenTelemetry | Chris Detsicas | 32m |
ThousandEyes describes a concrete pipeline: Zipkin from Istio/EngineX plus OTLP from apps flow through an OpenTelemetry Collector (Kubernetes operator) into… | ||
| Using Statistical Techniques to Automatically Detect Game-Breaking Issues | Ian Neidel | 15m |
Netflix builds an hourly pipeline that collapses 20M+ distinct error lines into 135 keys and 26 clusters using regex pre-processing, sentence-transformers… | ||
| Mapping a Better Future with STPA | Theo Klein | 19m |
Claims that applying Systems Theoretic Process Analysis (STPA) to Google Maps’ road-closure pipeline catches system-level design flaws that component-focused… | ||
| Is the S in SRE for “Security”? | John Benninghoff | 32m |
Argues that security performance tracks general engineering performance and cites three 2019 studies plus later meta-analyses: daily scanning teams close… | ||
| Lies Programmers Believe about Memory | Chris Down | 41m |
Claim: malloc doesn't allocate RAM — Linux overcommits and uses demand paging so physical pages appear only on page faults. Freeing often only returns memory… | ||
| “On-Call Is Ruining My Life” and Other Tales about Holding the Pager as an SRE | Cory Watson | 35m |
A 65‑question survey of largely 500+ org engineers with ~18 years’ experience finds 87% unhappy with on‑call status quo and 74% experiencing overload or… | ||
| SRE & Complexification: Where Verbs and Nouns Do Battle | David Woods | 41m |
Claim: marrying raw compute with richer representations lets SRE see ahead, see around and see anew to outmaneuver fresh “complexification” penalties from… | ||
| The Perverse Incentives of Reliability | Katie Wilde | 27m |
Claim: companies only reward reliability after SEV1s, so focus on keeping restorations shorter than the downtime tolerance window rather than chasing… | ||
| Maturing Your Data Architecture in a Week: How Bluesky Survived | Jaz Volpert | 39m |
Bluesky scales 10x in weeks to 33M users and peaks near 50k API requests/s and ~1,500 write events/s on bare‑metal across two data centers, forcing an 11‑day… | ||
| Inclusive SRE: Best Practices for Working with a Visually Impaired Incident... | Randall (Randy) Horwitz | 32m |
Shows a live PowerShell terminal demo running git add/commit/push with a hardware security key, demonstrating how a screen reader announces prompts and command… | ||
| Learning from Incidents at Scale; Actually Doing Cross-Incident Analysis | Vanessa Huerta Granda | 37m |
A four-person resiliency team runs blame-aware reviews for every major and sensitive incident, feeds an incident response bot into Jira incident reports, and… | ||
| Running DRP Tabletop Exercises | Josh Simon | 38m |
A college IT group runs collaborative tabletop exercises in Feb–Mar 2024 with a 15-person team, walking three scenarios: VM hypervisor compromise, a web host… | ||
| Optimizing Machine Learning Training Infrastructure: A Governance Approach | Anamaya Sullerey, Brian Hansen | 32m |
Meta describes a distributed hierarchical policy framework that enforces training workload governance via distributed hierarchical policy trees implemented as… | ||
| Beyond Sequential: A Recipe for Async Pipeline Observability and Alerting | Jash Mistry, Gabriela Medvetska | 36m |
eBay shows a concrete recipe: instrument async pipelines with Prometheus counters and histograms, mark each event state as success / retry / abandoned and… | ||
| Handling the Largest Domains Migration, Ever! | Franklin Angulo, Divya Kamat | 37m |
Squarespace acquires Google Domains assets and builds a migration pipeline that moves 9+ million domains into a new registrar supporting 360+ TLDs, a reseller… | ||
| Taming the Beast: Understanding and Harnessing the Power of HTTP Proxies | Guillaume Quintard | 33m |
Shows concrete fixes using Varnish: on‑the‑fly compression to halve cached size for text, a VCL ACL bug where reverse proxies accidentally authorize purge… | ||
| Chaos Experiments - Datacenter Stress Testing | Clayton Krueger | 46m |
A financial-services SRE program automates full single-data-center failover via a CI/CD pipeline with pre-checks, an SLO management tool that enables SLO… | ||
| Measuring Availability the Player Focused Way: How Riot Games Changed Its... | Maxfield F Stewart | 41m |
Riot defines availability as percentage of served player minutes using CCU data and incident records, builds ETL pipelines and Tableau heatmaps to roll… | ||
| Please Give Me Back My Network Cables! On Networking Limits in AWS | Steffen Gebert, Miklos Tirpak | 37m |
AWS EC2 enforces both bandwidth and packets‑per‑second caps so a “15 Gbps” instance is also limited to 1.25M packets/sec, meaning small IoT packets (150–200… | ||
| OpenTelemetry Semantic Conventions and How to Avoid Broken Observability | Dinesh Gurumurthy, Laurent Querel | 32m |
Semantic changes to OpenTelemetry (e.g. deployment.environment → deployment.environment.name, HTTP.method → http.request.method) silently break pipelines and… | ||
| Lightning Talks | Lauren Caliolio, Paige Cruz, Hamed Silatani, Clint Byrum, Kartheek Mahamkali, Will Gallego, Corey Quinn | 29m |
Seven seven mini-stories deliver concrete takeaways not platitudes: quitting tech to train as an EMT teaches that hands‑on emergency work builds confidence for… | ||
| Stopping Performance Regression via Changepoint Detection | Joseph Cirella, Shanthini Velan | 17m |
Bloomberg applies Bayesian online change point detection to latency time series and finds a multi-day regression that spreads across machines by comparing… | ||
| Per Aspera ad Productum: Turning Processes into Products | Yuri Bernstein | 22m |
Argues that SRE throughput rises when manual workflows become products: break processes into five reusable steps and build an interface that ties them… | ||
| Incident Management Metrics That Matter | Laura de Vesine, Jamie Luck | 37m |
Argues that MTTR and raw incident counts mislead because they’re noisy and create perverse incentives; instead shows concrete metrics that helped one SRE org… | ||
| Fully Automated HW SKU Selection System to Optimize Apache Pinot’s Cost-to... | Jia Guo, Yifan (Sabrina) Zhao, Dino Occhialini | 30m |
Claims a 35% reduction in cost-to-serve by migrating 20% of clusters and a 57% cut to future hardware forecasts, after aiming for ~20% savings; builds an… | ||
| Production Engineering When Trading Billions of Dollars a Day | Pedro Flemming | 47m |
Shows concrete incidents and numbers: Knight Capital 2012 burns $460M in 45 minutes and a 97‑email alert storm, a 2005 Tokyo typo sells 600,000 shares at ¥1… | ||
| Systems Thinking with Poisoned Systems | Hazel Weakly, Sandeep Kanabar | 37m |
Argues that observability AI learns patterns not systems and fails when training data is data poisoning or missing; gives concrete failures: AI hallucinated… | ||
| No Time to Do It All! Approaching Overload on DevOps Teams | Alex Wise | 33m |
Models a single service with a 4-engineer model writing 150,000 LOC, with bugs fixed in 1 hour if the author remains or 10 hours if not, showing backlog rising… | ||
| Securing Distributed Cache: Achieving Secure-by-Default with Key Challenges &... | Akashdeep Goel, Sriram Rangarajan, Samuel Fu | 38m |
Netflix runs an in‑house distributed cache serving 400 million requests per second across 200 clusters and 22,000 servers holding 14.3 petabytes of data; the… | ||
| Cattle vs. Pets - A Cost-Effective Elasticsearch Architecture to Scale-Out... | Leonardo Antônio dos Santos | 43m |
Managing tens of petabytes and trillions of logs monthly, the architecture moves from single massive clusters to a “cluster of clusters” with a centralized… | ||
| One Million Builds per Year, Only One Page - Operating Internal Services... | Cail Young | 15m |
Octopus Deploy’s small internal tools team runs the CI/CD platform that delivers 1.5 million builds per year while recording one out-of-hours page in a year by… | ||
| Going Multi Cloud in a Hurry with Quality and Style | Geoff Oakham | 15m |
A GKE-based managed Kubernetes platform adds AWS by standing up an EKS cluster, reusing Argo CD and Helm, and implementing Prometheus, Grafana, L7 load… | ||
| Mitigating Against Large Scale Systemic Failures in E-Trading | Chris Hawley | 11m |
Availability depends on time of day: an outage at market open causes outsized P&L, regulatory and reputational damage. Real incidents include London and US… | ||
| Hijacking Service Discovery to Simulate Dependency Degradation | Abdulrahman Alhamali | 16m |
Shopify runs 170M RPM average and built tooling to simulate degraded dependencies without changing service code: a fault-injection proxy Toxyroxy, a Kubernetes… | ||
| Network Flow Data in the Cloud | Steve Dodd | 16m |
Builds per-service flow maps from sFlow + hsfd agent sampled on every host, parses to pmacct -> Parquet to S3 and annotates with CI data to attribute traffic… | ||
| OLTP SQL Database Query Tracing and Linting | Xiaotong Jiang | 15m |
Databricks builds client-to-server query tracing that tags queries so operations can aggregate by tenant, RPC, API or error codes and quickly pinpoint which… | ||
| “How’s the App Doing?” Bringing Mobile Into Your Reliability Picture | Hanson Ho, David Rifkin | 17m |
Argues that server-side SLOs miss a huge slice of failures unless apps send user-centric telemetry and shows concrete gaps: requests stuck in client queues… | ||
| From HAR to OpenTelemetry Trace: Redefining Browser Observability | Antonio Jimenez | 18m |
Describes a pipeline that converts HTTP Archive JSON into OpenTelemetry traces, batches and enriches spans, and routes them via an OpenTelemetry collector… | ||
| Technical Debt as Theory Building and Practice | Yvonne Z. Lam | 51m |
Argues that treating technical debt like housework metaphor gives concrete handles: map the shape, pick bite-sized pieces, and craft a clear narrative. Gives… | ||
| AIOps: Prove It! An Open Letter to Vendors Selling AI for SREs | Charity Majors, Fred Hebert | 32m |
Vendors selling AI for SREs must show data, failure modes, and exactly what was built and tested — not slogans like “your new SRE.” The talk demands concrete… | ||
| nothing matches | ||