SREcon19 Europe/Middle East/Africa
2019-10-02..2019-10-04
Dublin, Ireland
who it's for
site reliability engineers, engineering managers, platform engineers, observability engineers, incident responders, support engineers, developer productivity engineers
topics
SLOs for data-intensive services; latency SLOs; building a scalable monitoring system; on-call rotations and tooling; support operations engineering; workplace mental health and burnout; PID control for autoscaling (Prometheus); eventually consistent service discovery (Envoy); eBPF network monitoring for TCP; HDR histograms and DDSketch summaries; tracing real-time distributed systems; fault tree analysis for Apache Kafka
| title | author | length |
|---|---|---|
| The SRE I Aspire to Be | Yaniv Aknin | 43m |
Argues that SRE work is measurement-first: pick a concrete reliability unit, trade reliability versus cost using error budget math, and move the… | ||
| A Systems Approach to Safety and Cybersecurity | Nancy Leveson | 46m |
Presents safety and security as control problems solved top-down with the STAMP/STPA method instead of component-failure thinking; shows concrete fixes… | ||
| SLOs for Data-Intensive Services | Yoann Fouquet | 24m |
Booking.com monitors a reservation pipeline that handles 1.5M reservations/day and 23 years of terabytes of data. SLOs evolve from naive HTTP success/latency… | ||
| Latency SLOs Done Right | Heinrich Hartmann | 27m |
Claim: percentiles alone mislead when aggregating across nodes and long windows, producing huge errors (example: ~68% error from naive averaging). Practical… | ||
| Building a Scalable Monitoring System | Molly Struve | 27m |
Company consolidates six disparate monitors (New Relic, Honeybadger, PagerDuty, LastAlert, StatusCake plus cron/dashboard checks) into Datadog as a single… | ||
| A Tale of Two Rotations Building a Humane & Effective On Call | Nick Lee | 28m |
Compares two rotations: a core ring zero rotation that reliably mitigates high-impact failures in ~30 minutes and has a training waitlist, versus a… | ||
| Support Operations Engineering: Scaling Developer Products to the Millions | Junade Ali | 23m |
Cloudflare scales support by turning diagnostics into engineering: a pipeline that fuses internal logs, a helper bot layer and machine learning to auto-resolve… | ||
| The Unmonitored Failure Domain: Mental Health | Jaime Woo | 35m |
Work-related mental health spreads between coworkers and correlates with measurable harms: nearly 1 in 4 workers aged 18–35 report daily stress, 1 in 6 say… | ||
| Control Theory for SRE | Ted Hahn, Mark P Hahn | 26m |
Builds a working PID controller that autoscaling Kubernetes can use and shows what actually happens: a simulated vehicle uses location as the process variable… | ||
| Eventually Consistent Service Discovery | Suhail Patel | 25m |
Argues that service discovery usually tolerates eventual consistency and that Monzo implements an Envoy sidecar model with a config provider backed by… | ||
| Network Monitor: A Tale of ACKnowledging an Observability Gap | Jason Gedge | 20m |
A Redis connectivity incident during peak sales triggers six weeks debugging and exposes an observability blind spot after engineers spot high TCP retransmits… | ||
| Being Reasonable about SRE | Vítek Urbanec | 24m |
Companies rename ops to SRE and expect instant magic, but the claim is that without analysis this creates higher expectations for unchanged teams and lots of… | ||
| From Nothing to SRE: Practical Guidance on Implementing SRE in Smaller Organisations | Matthew Huxtable | 26m |
Built SRE from zero inside a ~30‑engineer edtech company, growing an SRE team to 2–4 people and discovering a sustainable on‑call needs ~8 people (~40% of… | ||
| My Life as a Solo SRE | Brian Murphy | 26m |
2015 shows repeated outages, bad releases and frantic paging so the company creates a solo SRE role in 2016 and applies concrete changes: defines SLIs and… | ||
| Zero Touch Prod: Towards Safer and More Secure Production Environments | Michał Czapiński, Rainer Wolafka | 39m |
Zero Touch Prod enforces that every production change is automation-validated or goes through an audited break‑glass path; Google reports building a… | ||
| Zero-Downtime Rebalancing and Data Migration of a Mature Multi-Shard Platform | Justin Li, Florian Weingarten | 39m |
Shopify describes a production toolkit that uses the MySQL binlog event stream to copy and replay row changes, rebalancing hundreds of MySQL shards and… | ||
| All of Our ML Ideas Are Bad (and We Should Feel Bad) | Todd Underwood | 39m |
Most production ML proposals fail because the data and signal aren’t there: ticket prioritization and root‑cause models lack reliable labels, suffer delayed… | ||
| Fast, Available, Catastrophically Failing? Safely Avoiding Behavioral Incidents in Complex Production Systems | Ramin Keene | 30m |
Argues that model and data services fail as behavioral outages where bad data, not code, causes catastrophic user-visible behavior such as the high‑profile… | ||
| Advanced Napkin Math: Estimating System Performance from First Principles | Simon Eskildsen | 41m |
Shows how to turn guesses into numbers: memorize base rates (network, SSD, RAM, CPU at $10 per CPU) and combine them to decide if designs work. Measured… | ||
| The Map Is Not the Territory: How SLOs Lead Us Astray, and What We Can Do about It | Narayan Desai | 45m |
SLOs rest on shaky assumptions—independent requests, uniform request value and evenly distributed errors—that break on modern microservice stacks and low‑QPS… | ||
| Deploying SRE Training Best Practices to Production: How We SRE'ed Our SRE Education Program | Jennifer Petoff, JC van Winkel | 42m |
Google’s SRE EDU Orientation replaces lectures with a rigged production system: a breakable photo‑upload app (frontend, logic server, Cloud Storage, Spanner)… | ||
| SRE by Influence, Not Authority: How the New York Times Prepares for Large Scale Events | Vinessa Wan, Brett Haranin | 41m |
Large-scale elections readiness moves from mandates to partnership: over eight weeks (effectively six) more than 20 teams run architecture reviews, implement… | ||
| Load Balancing Building Blocks | Kyle Lexmond | 39m |
Facebook runs a seven-layered chain of load balancers per request (DNS → POPs → L4 → L7 → backends) and uses DNS steering plus a tool called cartographer to… | ||
| What Happens When You Type en.wikipedia.org? | Effie Mouzeli, Alexandros Kosiaris | 37m |
A concrete walkthrough of Wikipedia’s stack: MediaWiki on PHP 7.2 with four cache layers feeding Varnish/NGINX at the edge, migrating to ATS and Kubernetes… | ||
| Are We All on the Same Page? Let's Fix That | Luis Mineiro | 39m |
Adaptive Paging uses OpenTracing (soon OpenTelemetry) to read causal spans, grab exemplars for a breached symptom like "place order", walk child spans for… | ||
| Weathering the Storm: How Early Warnings Save the Farm | Brian Sherwin | 37m |
LinkedIn builds an automated correlation engine that ingests thousands of per-endpoint sensors, call trees/traces, metrics and deployment events, precomputes… | ||
| Refining Systems Data without Losing Fidelity | Liz Fong Jones | 35m |
Observability at production scale is unaffordable if every event is stored: a 99.95% SLO produces ~2,000× more “good” events than failures. The concrete plan… | ||
| Tracing Real-Time Distributed Systems | Evgeny Yakimov | 41m |
Bloomberg traces real-time trading pipelines by embedding a span-based tracer in the messaging library, sending data to a custom backend while reusing Jaeger’s… | ||
| How to Do SRE When You Have No SRE | Joan O'Callaghan | 28m |
Recommend committing one to two hours a week to SRE work, running quick reviews that document worst-case failure modes, capacity headroom, access lists… | ||
| One on One SRE | Amy Tobey | 42m |
A one-on-one incident debrief process runs 30–60 minute Zoom interviews using a fixed question set and real‑time notes in a Google Doc that are shared only… | ||
| A Customer Service Approach to SRE | John P. Looney | 40m |
Facebook's Server Lifecycle team shows concrete changes: they add installer options to run burn‑in scripts and disable components so technicians stop… | ||
| SRE & Product Management: How to Level up Your Team (and Career!) by Thinking like a Product Manager | Jen Wohlner | 37m |
SRE teams adopt five concrete practices: run 30‑minute user interviews with diverse roles instead of assuming workflows, ask non‑leading questions, and run… | ||
| Prioritizing Trust While Creating Applications | Jennifer Davis | 38m |
Argues that security must shift security left into design and CI/CD by threat modeling, tagging security stories, and feeding findings back into pipelines… | ||
| Software Patching Needn't Be a Can of Worms | Philip Rowlands | 30m |
Argues that automation replaces tedious manual patching by doing four repeatable steps: inventory, assess and classify, plan rollout and rollback, then repeat… | ||
| Lightning Talks | Igor Ebner de Carvalho, Laura Nolan, Amir Langer, Doron Sekler, Susobhit Panigrahi, Yannick Brosseau, James Renken, Brennen Smith, Jane Maguire, James Wynne | 46m |
Ten rapid case studies: a lemonade-as-a-service model shows 100% availability drives exploding cost (machines raise price to €1.12, need N+2); causal loop… | ||
| Bigtable: A Journey from Binary to Service and the Lessons Learned along the Way | Brendan Gleason, Gaurav Prabhu Gaonkar | 38m |
Bigtable evolves from per-cluster binaries into a global multi-tenant service that runs for over a decade and powers 60+ internal projects, with tables that… | ||
| SDKs Are Not Services and What This Means for SREs | Justin Coffey | 41m |
Criteo builds an embedded SDK scheduler in Scala to orchestrate hourly Hadoop/MapReduce and Hive jobs across a platform that sees ~1+ billion monthly users and… | ||
| Building Resilience: How to Learn More from Incidents | Nick Stenning | 40m |
Research-backed claim: incidents reveal both prevention gaps and response weaknesses, so investigate design and context, not blame. Shows the B-17 case where… | ||
| How Stripe Invests in Technical Infrastructure | Will Larson | 43m |
Defines infrastructure as any tool used by three-plus teams for business-critical work and gives concrete tactics: shard MongoDB, script-kill long queries, add… | ||
| Why Automating Everything Adds to Your Toil | Colin Thorne, Cameron McCallister | 38m |
Claim: blanket automation increases long‑term toil because scripts rot, dependencies change (ServiceNow, EU cloud rules, OS upgrades) and duplicated tools pile… | ||
| Autopsy of a MySQL Automation Disaster | Jean-François Gagné | 41m |
Automated MySQL master failover using Orchestrator promotes an intermediate node X after two masters (A and B) go down, then a separate process — a DNS… | ||
| Pushing through Friction | Dan Na | 35m |
Explores why organizational friction stalls a 75% improvement that would take two weeks, ties recurring failures to the normalization of deviance, and frames… | ||
| Perks and Pitfalls of Building a Remote First Team | Ryan Neal | 39m |
Company grows to ~60 people with about 60% remote, an 11‑person umbrella team spanning four time zones and five countries, and uses Donut plus Slack channels… | ||
| Expect the Unexpected: Preparing SRE Teams for Responding to Novel Failures | John Arthorne | 25m |
Shopify reports $1,700 USD/sec at stake and an SRE org grown from ~30 to ~120; preparation emphasizes specific practices: open incident channels (Slack bots… | ||
| Hiring Great SREs | Brian Rutkin | 23m |
Claims that personality traits predict SRE success more than listed tech skills, and that a structured, repeatable pipeline yields clearer hiring decisions… | ||
| SRE in the Third Age | Björn Rabenstein | 25m |
Frame: SRE evolves from a Google-only practice to a public discipline and now to a diffuse skill set where most engineers must carry SRE mindset. Small… | ||
| Evolution of Observability Tools at Pinterest | Naoman Abbas | 42m |
Pinterest moves its metrics backend from Graphite to OpenTSDB to ten sharded clusters and then to an in‑memory store called Goku (stores 24h with Gorilla… | ||
| How to SRE When Everything's Already on Fire | Alex Hidalgo, Alex Lee | 40m |
Squarespace fixes a months‑long ELK outage by turning vague alerts into a single consumer‑lag / write‑rate SLI, raising SLOs from “5 min 99%” to “2 min 99.9%”… | ||
| Fault Tree Analysis Applied to Apache Kafka | Andrey Falko | 38m |
Provides numerical fault-tree answers for Kafka choices: uses AFR disk rates (rotational 4%, SSD 0.8%), network 2%, hardware/OS 1%, and remediation windows to… | ||
| Applicable and Achievable Formal Verification | Heidy Khlaaf | 33m |
Explains how industry applies formal methods to real embedded devices: mandates from IEC 61508 map verification techniques to Safety Integrity Levels, and a… | ||
| nothing matches | ||