SREcon20 Americas
2020-12-07..2020-12-09
-- online --
who it's for
site reliability engineers, engineering managers, platform engineers, observability engineers, incident response leads, devops engineers, security engineers
topics
service level objectives and error budgets; distributed tracing for dependency diagrams; chaos engineering for stateful systems; Jenkins on Kubernetes CI/CD; session replay and customer telemetry; pager fatigue and on-call culture; mutual TLS for pod-to-pod communication; heap optimization for Go systems; hot swap datastore strategies; implementing distributed consensus; PXE bootstrapping and provisioning; capacity planning with page reference sampling
| title | author | length |
|---|---|---|
| The Secret Lives of SREs - Controlling the Costs of Coordination across Remote | Laura Maguire | 48m |
Researcher reports analysis of 62 cases across 4 organizations showing that costly coordination, not just technical failure, drives incident complexity; finds… | ||
| Identifying Hidden Dependencies | Liz Fong Jones | 34m |
Honeycomb deliberately terminates nodes on purpose to test Zookeeper, Kafka, and indexing durability, running controlled restarts every Monday at 3 PM and… | ||
| Are We Getting Better Yet? Progress Toward Safer Operations | Alex Elman | 41m |
Argues that counting incidents or MTTR obscures progress and pushes for a shift from prevent-and-fix to a learn-and-adapt approach, showing concrete artifacts… | ||
| Continuously Improving Culture through Design Decisions | Zachary Meath | 39m |
Two Sigma builds COIN, a CI/CD platform that centralizes Jenkins on Kubernetes and drives culture through design: Jenkins on Kubernetes, 90+ Jenkins instances… | ||
| Avoiding Goodhart's Law - Use SLO's as Tools Not Cudgels | Marco Coulter | 34m |
Hospital lab system story shows a real Goodhart failure: a monitor for queue length >100 becomes the target while transactions time out before the queue, so… | ||
| Off the Beaten Path: Moving Observability Focus from Your Service | Mohit Suley | 38m |
Argues that observability must shift toward measurable customer experience and gives concrete techniques: use W3C Network Error Logging to expose survivorship… | ||
| Observing from Incidents | Cory Watson | 43m |
Companies mine Slack incident transcripts with Jelly to extract the concrete questions people ask and then encode those signals into dashboards: link runbooks… | ||
| Building Service Ownership Using Documentation, Telemetry, and a Chance to Make | Daniel Spoonhower | 38m |
Company uses services.yaml as the single source of truth to generate dashboards, escalation policies and CI/CD entries, and it augments human-written wikis… | ||
| Study on Human Factors and Team Culture to Improve Pager Fatigue | Daria Barteneva | 19m |
A study across seven months uses surveys and interviews to show the number of pages doesn't correlate with on-call satisfaction; instead, teams report higher… | ||
| Squish Level Objectives: How SRE can Help Align Technical Work to User Benefit | Dave Stanke | 14m |
Argues that reliability is a customer-facing feature and that SLOs and Error Budgets are the concrete tools to align engineering work to user benefit, urging… | ||
| Building Actionable Code Ownership | Anika Mukherji | 15m |
Pinterest codifies ownership for API endpoints so each request maps to a responsible project, not an individual team, and applies that to over 1,700 endpoints… | ||
| SREs at Telecom and Media Industry: Bridging between Legacy and Cloud Native App | Shiri Yitzhaki, Itzhak Tueg, Tapan Shah | 15m |
Amdocs implements SRE across a 5,000-person managed services unit called Smart Ops using an atomic automation platform and inner sourcing to share work… | ||
| Jupyter as Incident Response Tool | Moshe Zadka | 20m |
Jupyter is used as an incident response console that queries Prometheus for five minutes of data at 10s resolution, finds the largest amplitude among four… | ||
| Sustainable Software Engineering & SREs | Bill Johnson | 19m |
Planetary SLO sits at 1.5°C and current warming is about 1.15°C with roughly 12 years before breach; software choices affect that. Examples include cutting… | ||
| When /bin/sh Attacks: Revisiting "Automate All the Things" | J. Paul Reed | 41m |
Argues that pervasive shell automation and orchestration (bash, Jenkins, Slack low‑code) can degrade operator skills and hide failure modes, citing Knight… | ||
| Testing Encyclopedias in Production | Effie Mouzeli, Giuseppe Lavagetto | 36m |
Wikimedia migrates from HHVM/PHP5 to php-fpm and PHP7 serving ~100k req/s and ~22 billion page views monthly, using a gradual cookie-based rollout that reaches… | ||
| Why SREs can't afford to NOT do Chaos Engineering | Miko Pawlikowski | 39m |
Claims Chaos Engineering is a scientific method to build measurable confidence and works at any scale. First, attach strace and inject write syscall failures… | ||
| Implementing Distributed Consensus | Dan Lüdtke | 40m |
A minimal educational lock service called Skinny implements a Paxos-like protocol in Go using protocol buffers and gRPC, running five replicas across… | ||
| Incident Response in Unfamiliar Sociotechnical Systems | Morgan Collins | 40m |
Adoption of the Incident Command System since 2012 at Salesforce is credited with measurable improvements in incident response, but the pandemic creates uneven… | ||
| Making Infrastructure More Friendly for Beginners | Shubheksha Jalan, Leslie Carr | 39m |
Career path shows a concrete route from front-end into cloud-native infrastructure: learning Go and Kubernetes, contributing to projects like Kubernetes and… | ||
| Low Context DevOps: Improving SRE Team Culture through Defaults, Documentation | Tom Limoncelli | 33m |
Argues SRE teams must be carefully constructed defaults, make right easy, and ubiquitous documentation so newcomers and on-call responders aren't blocked… | ||
| The Smallest Possible SRE Team | Zach Thomas | 38m |
Argues a single-person SRE can be effective by directing data, tools, and communication toward teams and amplifying impact with automation and docs. Runs… | ||
| Cloudy with a Chance of Chaos | Christina Yakomin | 38m |
Vanguard builds a homegrown chaos platform called Climate of Chaos that runs experiments named Cyclone/Tornado/Blizzard and now supports 16 fault types… | ||
| Confessions of a Systems Engineer: Learning from My 20+ Years of Failure | David Argent | 39m |
No change is safe: a config meant for test once reaches prod via a VIP and takes out prod services, so every rollout uses canary under deployment with staged… | ||
| Pragmatic Security for SRE | James Wickett | 39m |
Pragmatic approach maps security controls to five pipeline phases—develop, inherit, build, deploy, operate—and gives concrete tools and checks: pre-commit… | ||
| "Disorganizing" Your SRE Organization | Leonid Belkind | 38m |
After COVID forces remote work the SRE role is democratized across engineers and managers, with reliability treated as a feature and on-call responsibility… | ||
| Failure is Not an Option! SRE Lessons 50 Years after the Apollo 13 Flight | Robert Barron | 39m |
An oxygen tank explosion cripples the service module, leaving no power or fuel; the Saturn V’s instrument unit automatically extends a stage burn to compensate… | ||
| Challenges of Starting an SRE Team from Scratch in an Enterprise | Pauline P. Narvas, Wayne Bridgman, Graeme Bye, Amreen Firdouse, Anand Bobade, Shiv Patil | 33m |
Large enterprise (100k+ employees) starts SRE work in March 2020 by reallocating existing engineers rather than hiring, creates BT Sentinel for automated… | ||
| The Good, the Bad and the Ugly: The 3 Learnings of an SRE | Prathyusha Charagondla | 18m |
Three concrete learnings: build tooling to reduce MTTD/MTTR (automated rollbacks, detection playbooks) and run controlled failures with chaos mesh / Chaos… | ||
| 9 Years of Failure: How Racing Crappy Cars Made Me a Better SRE | Ryan Doherty | 14m |
Nine years of 24 Hours of Lemons racing yields concrete SRE lessons: teams of 4–6 maintain $500 cars, survive ~half a dozen engine swaps and a center‑support… | ||
| Production Population Control: My Cattle are Rabbits! | Alex Nauda | 18m |
Cloud environments spawn many more environments than expected—often 6–9 test stacks per app and sometimes up to 10 production regional/DR variants—so define… | ||
| Latency and Availability Error Budgets Done Right at Scale | Fred Moyer | 19m |
Zendesk codifies SLI → SLO → error budget as simple math: metric identifier operator metric value and success objective over a time window, then error budget =… | ||
| The Evolution of Traffic Routing in a Streaming World | Abhishek Srikanth | 21m |
A team migrates from co‑located services to an off‑box fleet and then to a full‑mesh where every gateway connects to every Blade Runner. They solve connection… | ||
| Heap Optimization for Go Systems | Nishant Roy | 19m |
GODEBUG=gctrace=1 plus runtime.MemStats and pprof heap profiles identify GC spikes and show whether p99 latency correlates with collections. Replacing string… | ||
| Soft Failures, Hard Goals - Accelerating Payments at Scale During the Pandemic | Kyle Guichard, Venus So | 38m |
A 90-day resilience project timeboxes six workstreams—alerts and metrics, reconciliation, test automation, anti-fragile design, education and automation—and… | ||
| Give Your PXE wings! Bootstrapping Explained | Rob Hirschfeld | 39m |
Explains PXE as a multi-stage network bootstrap and shows a working stack: Digital Rebar plus a sledgehammer RAM-only OS for fast discovery, iPXE for HTTP… | ||
| Hot Swap Your Datastore: A Practical Approach and Lessons Learned | Raj Shekhar, Mehmet Can Kurt | 36m |
Quantcast migrates a critical cookie datastore from keebler to aerospike with zero production downtime while keeping 99th‑percentile lookup latency near the… | ||
| Automatically Detect the Top Performance & Scalability Issues in Distributed | Andreas Grabner | 37m |
Claims automated detection of common distributed anti-patterns across 100+ teams in 2019 and names the offenders: N+1 call & query, payload flood, tight… | ||
| A Bartender's Guide to Network Monitoring | John Blaho | 37m |
Claims that SLOs+SLIs drive vendor accountability, that an outside-in DEM approach finds real user pain, and that monitoring must include DNS/BGP, CDN edges… | ||
| Achieving the Ultimate Performance with KVM | Boyan Krosnov | 40m |
Shows concrete knobs and measured results: use NVMe + StorPool and vhost-user with OVS+DPDK or SR‑IOV for lowest network latency (measured ~14µs vs ~37µs for… | ||
| Weeks of Debugging Can Save You Hours of TLA+ | Markus A Kuppe | 39m |
A TLA+ spec of a bounded MPMC queue reproduces and explains a surprising deadlock that unit tests miss: TLC finds a violating trace (shortest 9 steps, another… | ||
| Capacity Planning and Performance Enhancement with Page Reference Sampling | Danny Chen | 41m |
Shows how Solaris /proc page data yields per-page read/write sampling without kernel patches and introduces two tools: wssts (periodic working-set counts) and… | ||
| Achieving Mutual TLS: Secure Pod-to-Pod Communication Without the Hassle | Mark P Hahn, Thomas Edward Hahn | 32m |
Claims a Kubernetes admission controller that auto-creates per-pod X.509 keys and CSR objects, approves them, and mounts signed certificates as Secrets so apps… | ||
| Panel: Learning from Adaptations to Coronavirus | Fred Hebert, Lorin Hochstein, Vanessa Huerta Granda, Nora Jones | 51m |
Five themes emerge from two facilitated calls and three interviews with engineers at Postmates Netflix and Enova: remote war rooms move to Google Meet and… | ||
| It's a Trap! How Abstractions Have Failed Us. | André Henry | 38m |
Argues that adding layers like Kubernetes and containers often trades short-term productivity for long-term cost: lost hardware proficiency, brittle debugging… | ||
| nothing matches | ||