SREcon19 Americas
2019-03-25..2019-03-27
Brooklyn, United States
| title | author | length |
|---|---|---|
| What Breaks Our Systems: A Taxonomy of Black Swans | Laura Nolan | 31m |
Catalog of real failure modes: hidden resource caps like ext3 2TB limit that force a dump-and-reimport to ext4, logical counters such as Postgres xid wrap that… | ||
| Complexity: The Crucial Ingredient in Your Kitchen | Casey Rosenthal | 29m |
Draws a direct line from the Challenger O‑ring failure to microservice failures: a cache hides read pressure until Cassandra slows and Cassandra returning a… | ||
| Case Study: Implementing SLOs for a New Service | Arnaud Lawson | 20m |
Squarespace's Ceph-backed S3-compatible object store implements SLOs using synthetic probers instead of instrumenting every client, collects four weeks of SLIs… | ||
| Fixing On-Call When Nobody Thinks It's (Too) Broken | Tony Lykke | 28m |
Hudson River Trading reduces paging from ~200/week (~70k over 7 years) to under 50/week in months by data-driven fixes: import PagerDuty history into… | ||
| Lessons Learned in Black Box Monitoring 25,000 Endpoints and Proving the SRE Team's Value | Aaron Wieczorek | 23m |
Built a black‑box monitor for 25,000+ .gov domains that records HTTP status and latency, starting with a Python GET-based MVP and scaling to Prometheus +… | ||
| Keeping the Balance: Internet-Scale Loadbalancing Demystified | Laura Nolan, Murali Suriar | 30m |
Starts from a single A record and shows why DNS multi‑IP and short TTLs still fail for health and control, then moves to network load balancers using 5-tuple… | ||
| Aperture: A Non-Cooperative, Client-Side Load Balancing Algorithm | Ruben Oanta | 26m |
Twitter’s Finagle replaces random per-client windows with a ring-based continuous coordinate aperture that composes client-local orders and uses… | ||
| Capacity Prediction in External Services | Jerome Kraus | 28m |
Predicts vendor capacity by modeling utilization as a linear function of a load factor = Σ(request_rate × duration) and fits that model in Microsoft Excel… | ||
| How Did Things Go Right? Learning More from Incidents | Ryan Kitchens | 30m |
Argues SRE work shifts from chasing fixes to mapping how systems and teams succeed: use SLOs and error budgets for context, run chaos experiments until… | ||
| Code-Yellow: Helping Operations Top-Heavy Teams the Smart Way | Michael Kehoe, Todd Palino | 25m |
LinkedIn codifies Code Yellow as a time‑boxed repair process: traffic SRE gets five dedicated engineers for three months to rebuild monitoring, simplify the… | ||
| Creating a Code Review Culture | Johnathan Turner | 29m |
Explains concrete steps to build a code review culture: publish review values and create language and domain experts with pathways to certify them; use a PR… | ||
| Benefits of Taking the Less Traveled Road with Containers Infrastructure | Eduard Iacoboaia | 15m |
After almost a year on OpenShift Origin the team migrates to vanilla Kubernetes, citing a week-long outage during an upgrade and slow two‑week role… | ||
| The Ops in Serverless | Jennifer Davis | 15m |
Argues that serverless lowers provisioning work but raises operational complexity and so requires deliberate governance, CI/CD and instrumentation. Notes a… | ||
| Testing in Production at Scale | Amit Gud | 21m |
Argues that running testing in production at Uber scale (600 cities, 75M riders, 15M trips/day) is viable by attaching a tenancy context to every request and… | ||
| Tackling Kafka, with a Small Team | Jaren Glover | 28m |
Robinhood runs Kafka as a critical pipeline after a one-engineer team grows usage; topics hit replication factor 3, clusters ingest ~80 TB/week, and ad hoc ops… | ||
| Tracing, Fast and Slow: Digging into and Improving Your Web Service's | Lynn Root | 30m |
Practical checklist for tracing a web service: instrument with metadata propagation (Dapper-style) or fall back to simple UUID headers and log correlation… | ||
| Visibility into Loggers (and Other Low Level Services)—Seeing the Trees from the Forest | Danny Chen | 30m |
Measurements show file-based logging often costs 100–170 µs median with long tails out to 3.3 ms tail, while HTTP-backed logging moves medians into… | ||
| Operating within Normal Parameters: Monitoring Kubernetes | Elana Hashman | 30m |
Defines SLOs like 99% control-plane uptime and pod start p99 <5s, then builds a minimal in‑cluster monitoring stack using Prometheus, kube-state-metrics… | ||
| SRE & Product Management: How to Level up Your Team (and Career!) by Thinking like a Product Manager | Jen Wohlner | 30m |
SRE teams learn to think like product managers to make internal tools actually usable: run 30-minute user interviews with diverse coworkers, ask non‑leading… | ||
| Shipping Software with an SRE Mindset | Theo Schlossnagle | 27m |
Applies SRE practices to shipped on‑prem software: a productionized back-trace collector yields 300KB mini‑dumps ~35 ms after a crash so engineers get full… | ||
| Using PRDs and User Journeys to Design User-Friendly Tools | Gwendolyn Stockman | 23m |
Shows a concrete process for turning a feature idea into a PRD: build a persona-driven house-evaluation app and write a full user journey first (Abigail… | ||
| SRE Classroom - How to Design a Distributed System in 3 Hours | Ryan Thomas, JC van Winkel, Phillip Tischler, Jennifer Mace | 01h 03m |
Workshop converts a sketch into a runnable design: given a million users uploading 50 photos/day of 4 MB each and thumbnails of 256 KB, the exercise uses the… | ||
| Migrating a Monolith to the Cloud | Keyur Govande | 27m |
Etsy performs a lift-and-shift of a 14‑year monolith to GCP in roughly eight months, using terraform for infra-as-code and extensive pre-cutover testing that… | ||
| An Introduction to GraphQL | Nat Welch | 23m |
GraphQL is an application-layer query schema (Facebook origin, spec stable since 2018) where the request shape defines the JSON response shape and the schema… | ||
| Service Discovery Challenges at Scale | Ruslan Nigmatullin | 27m |
Dropbox describes a service discovery system named Sasanian that uses Zookeeper plus a per-host local daemon to multiplex clients and cache reads, coalesces… | ||
| Inside the Kube: A Guided Tour of Kubernetes Cluster Setup | Liz Frost | 01h 31m |
Walks through building a single-node Kubernetes cluster using kubeadm, the kubelet, Docker and a self-hosted etcd pod, generating all TLS certificates and… | ||
| What I Wish I Knew before Going On-call | Chie Shu, Dorothy Jung, Wenting Wang | 01h 25m |
Yelp ad platform teams report that about two thirds of new engineers feel unready for on‑call and it typically takes three to six months to ramp; they build a… | ||
| Running Excellent Retrospectives: Talking for Humans | Courtney Eckhardt, Lex Neva | 01h 30m |
Prescribes three facilitator duties: create a safe learning environment, run a focused meeting with a rotating note-taker, and avoid destructive humor… | ||
| Lightning Talks | Bridget Kromhout, Jaime Woo, Dawn Parzych, Paul Henry, Ravi Lachhman, Aniket Kulkarni, Anthony Sandoval, Matthew Fornaciari, David Holmes | 01h 09m |
Livetweeting Tech Conferences
Bridget Kromhout, Microsoft
5 Insights from 200 SREs on How Incident Response Affects Them
Jaime Woo, Dawn Parzych… | ||
| Optimizing for Learning | Logan McDonald | 27m |
Argues that the human mind is the best observability tool and that expert intuition is achievable by training memory and models rather than waiting for… | ||
| Zero to SRE | Kim Schlesinger | 29m |
Claims that a junior SRE becomes mid-level within one year by enforcing 20% time for learning, running 4–5 person pods, and using a public engineering leveling… | ||
| One on One SRE | Amy Tobey | 30m |
Describes two programs: 1:1 incident debrief and 1:1 SRE outreach that run as short guided interviews (markdown agenda, private docs) to collect personal… | ||
| The Curse of SRE Autonomy and How to Manage It | Richard Bondi | 28m |
A ten‑year internal scheduler called Sisyphus becomes widely adopted across SRE teams despite no roadmap, poor Python code, and leadership trying to retire it… | ||
| Learning from Learnings: Anatomy of Three Incidents | Randy Shoup | 30m |
An App Engine 8-hour global outage in October 2012 triggers a timeboxed, whiteboard-driven program: teams bucket eight-to-ten themes, spend one week scoping… | ||
| Fault Tree Analysis Applied to Apache Kafka | Andrey Falko | 31m |
Applies fault tree analysis to Kafka clusters and gives concrete numbers: assumes 4% annual disk failure, 2% network, 1% hardware and OS faults, computes a… | ||
| Strategies to Edit Production Data | Julie Qiu | 26m |
Five concrete strategies for making SQL writes safer: keep a shared spreadsheet for audit trail and peer review; run Python migration scripts locally with a… | ||
| Madaari: Ordering for the Monkeys | Ashutosh Raina, Ramprasad Ellupuru | 28m |
Claims that ordering LDFI finds the most likely failures first and prunes huge fault spaces using actual traces and SAT/ILP solving. eBay builds lineage from… | ||
| Sublinear Scaling in Practice: The 1k SRE Project | Nikolaus Rath | 27m |
About 40 people in two locations run roughly 400 Google Ads services and report growing from 187 to 431 services in three years without headcount increases… | ||
| Pragmatic Automation | Max Luebbe | 25m |
Starts from four Google regions and an SLO like “build a location in eight weeks,” then moves from human checklists to markdown with HTML comments parsed into… | ||
| Differences in SRE Implementations across Companies | Kurt Andersen | 30m |
Companies deploy embedded SREs or central horizontal pods and sometimes both; reporting ranges from direct CIO lines to being inside product engineering… | ||
| Latency SLOs Done Right | Fred Moyer | 31m |
Slides present three concrete methods: parsing logs (accurate but costly — about 100 bytes per request → 1 GB per 10M requests, often stored in… | ||
| Extending the Error Budget Model to Security and Feature Freshness | Jim Thomson, David Laing | 19m |
Extends error‑budget mechanics to concrete measures: SLI = days since a dependency patch, SLO = a max‑age (example 30 days), policy = apply patches when… | ||
| You Don't Have to Love Your Job | Leslie Carr | 15m |
Argues that people do not have to love their job to be good at it and that liking your work is sufficient, traces the myth to the Calvinist Protestant work… | ||
| Mindfulness in SRE: Monitoring and Alerting for One's Self | Tommy Lutz | 16m |
A single-engine flight goes engineless at 6,000 ft with an 800 ft/min descent from a 1,000 ft abyss and RPM dropping from ~2,500 to zero, triggering an… | ||
| Automating the Management of the Operational Health of Cloud Accounts at Scale | Jamie Walls | 31m |
Capital One runs Cloud Custodian across many AWS accounts plus custom Lambdas and a Dash monitoring app to enforce rules such as stop if CPU <1.5% over 14 days… | ||
| Designing Resilient Data Pipelines | Andrew Bolin | 30m |
Shows concrete fixes: collect logs plus pipeline- and task-level metrics to trace a days-long ETL latency jump to a degraded shared resource; encode pipelines… | ||
| From Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based | Salim Virji, Carlos Villavieja | 30m |
Sixty interviews across production teams yield five concrete rules: monitor model freshness because model quality degrades with age, stamp models with… | ||
| Resilience Engineering Mythbusting | Will Gallego | 30m |
Defines resilience as the system plus people adapting beyond expected bounds and separates it from reliability and robustness, shows chaos engineering is for… | ||
| Why Are Distributed Systems So Hard? | Denise Yu | 28m |
A clear account of why partitions are inevitable: networks fail hardware and software with examples like five rack failures, three router failures, and eight… | ||
| nothing matches | ||