SREcon23 Americas
2023-03-21..2023-03-23
Santa Clara, United States
who it's for
site reliability engineers, platform engineers, incident responders, SRE managers, observability engineers, engineering leaders
topics
OpenTelemetry metrics pipeline; distributed tracing futures; large-scale telemetry with Kafka; Terraform code preprocessor (Stacks); slo adoption and failure modes; incident analysis and narrative archaeology; cdn resiliency and regional failover; chaos-driven development for distributed systems; DNS and conntrack debugging; adaptive concurrency control for analytics; low-latency trading with Go and Java; botnet DDoS defense strategies
| title | author | length |
|---|---|---|
| The Endgame of SRE | Amy Tobey | 33m |
A production deploy includes a schema rename that schema migration fails mid-deploy; the CI/CD system rolls back but the half-applied change leaves another… | ||
| SRE's Critical Role in the COVID-19 Pandemic Response in Government | Amy Quispe, Marc Alvidrez, Rick Hawes | 37m |
Built and deployed government-grade services that actually moved data and goods: SimpleReport and ReportStream give ad-hoc testing sites an EHR and a… | ||
| We're Still Down: A Metastable Failure Tale | Kyle Lexmond | 25m |
A globally distributed CDN enters a metastable failure when a DR test plus cross‑region transcoder compression causes CPU spikes, OOMs and health checks to… | ||
| Watering the Roots of Resilience: Learning from Failure with Decision Trees | Kelly Shortridge | 41m |
Argues that aligning mental models with reality requires chaos experiments plus documented decision trees; shows an open‑source YAML tool (deciduous) to… | ||
| Scaling Telemetry Systems with Streaming | Liz Fong Jones, Terra Field | 41m |
Company telemetry grows from 250k to 2.5M events/sec so Kafka becomes the single ingest path; architecture uses ~100 partitions (~200 indexers) with ~50k… | ||
| Hacking the Pachyderm: Scaling Servers and People | Hazel Weakly, Preston Doster | 40m |
Community instance grows 700 to 30,000 users in weeks, triggers transatlantic migration of a distributed Mastodon stack with zero data loss. Ops details… | ||
| Logs Told Us It Was DNS, It Looked like DNS, It Had to Be DNS, It Wasn't DNS | Hemanth Malla, Elijah Andrews | 38m |
Kubernetes pods hit intermittent DNS failures during rolling restarts while node-local-dns OOMs despite increasing memory from 64→256 MiB and max_concurrent… | ||
| Epic Incidents of History: The 1979 NORAD Nuclear Near Miss | Nick Travaglini | 32m |
On 9 November 1979 a test reel and an NJ scenario tape reach NORAD s Honeywell 427M via the MGR and CSS while processors run in hot shadow mode a satellite… | ||
| Scaling Terraform at | Ricard Bejarano | 25m |
ThousandEyes moved from duplicate Terraform repos and 5,100 .tf files to a code-preprocessor called Stacks that keeps Terraform state sizes small while… | ||
| OpenTelemetry Metrics 101 | Reese Lee | 42m |
OpenTelemetry metrics are explained using a live telescopes demo app and the full metrics pipeline: API → SDK → metric reader → exporter → backend via OTLP and… | ||
| Incident Commanders to Incident Analysts: How We Got Here | Vanessa Huerta Granda, Emily Ruppe | 26m |
Argues incidents are a continuous incident lifecycle not one-off events, and that response and analysis require distinct skill sets; recommends separating… | ||
| Handover Communications in Software Operations: Findings from the Field | Chad Todd | 30m |
Research interviews across NOC and customer support find that purely written handovers reduce confidence, while slack workflows with a "knowledge" button and… | ||
| On the Wings of SREs; J.P. Morgan's Journey into the Cloud | Fred Moyer | 20m |
J.P. Morgan Chase creates an SRE function to catalyze cloud migration across 57,000 technologists and a $14B annual tech budget, moving from under 100 to over… | ||
| SRE in Transition: From Startup to Established Business | Laura de Vesine | 16m |
Maps specific growth pain to actions: documents common startup technical debt (shared DB credentials, inconsistent service discovery, 2–hour reshard… | ||
| Lessons Learned from 7 Years of Running Developer Platforms | Tony McCulley | 32m |
Seven years running a developer platform yields concrete moves: hire product and UX roles, run a value stream map to cut handoffs, ship a pre-integrated… | ||
| Cognitive Apprenticeship in Practice with Alert Triage Hour of Power | Paige Cruz | 32m |
A recurring one‑hour, twice‑monthly meeting teaches real triage from a PagerDuty page through traces, dashboards and runbooks using explicit roles… | ||
| Building a Diverse SRE Talent Pipeline | Salim Virji, Alex Gornet | 31m |
Google partners with Major League Hacking to run a 12-week fellowship that combines the Linux Foundation curriculum, hands-on projects and a mentor-led pod… | ||
| The Best SREs Seem to Be the Ones without an SRE Title—And What We Can Do | Kishore Jalleda | 35m |
Argument: most effective reliability work lives outside the org chart, shown by a single incident where a fiber cut cascades into service failures, broken… | ||
| Confessions of an SRE Manager | Andrew Hatch | 31m |
Argues that SRE managers must stay technical and visible: do on-call shifts to learn unplanned work and help burnt-out engineers, hold weekly 1:1s to build… | ||
| Exploring Disconnects between Reliability Practitioners and Management | Kurt Andersen, Leo Vasiliou | 34m |
Survey of 559 responses from Summer 2022 finds systematic perception gaps: AIOps value rated low by individual contributors vs executives who skew toward… | ||
| Beacon: Intelligent Latency-Aware and Load Shedding Service Routing | Jason Griggs, Huajun Qin | 30m |
Beacon is a service router that checks per-key last-update times against live replication delays and routes reads to either primary read-write or read-only… | ||
| Resiliency Practices in Managing CDN (Content Delivery Network) | Yeshwenth Jayaraman | 37m |
Netflix Open Connect runs ~18,000+ OCAs in ~6,000 locations across 175 countries with ~70 IX sites on a multi‑terabit backbone, and validates resiliency with… | ||
| Why This Stuff Is Hard | Lorin Hochstein | 30m |
Argues systems live in an Adaptive Universe: resources are finite and change never stops, so growth in capacity triggers the law of stretch systems and teams… | ||
| Turning an Incident Report into a Design Issue with TLA+ | A. Finn Hackett, Markus A Kuppe | 31m |
A precise TLA+ specification of Cosmos DB behavior reproduces the 28-day outage as a counterexample and shows the root cause is missing session tokens between… | ||
| The Making of an Ultra Low Latency Trading System with Go and Java | Yucong Sun, Jonathan Ting | 47m |
An exchange-grade matching engine serializes each market as a deterministic state machine and achieves sub-50 microseconds E2E with matching times near 1… | ||
| Seeing the Invisible: Two Years at Wikipedia with W3C's Network Error Logging | Chris Danis | 31m |
Wikipedia deploys W3C Network Error Logging across half of users and receives near-real-time browser error reports (often under a minute, sometimes 0 ms) via a… | ||
| Avoiding Cachepocalypse in the Land of the Monolith | David Amin | 12m |
Legacy Python monolith exhausts memcached capacity because one cache node receives duplicate connections, pushing a normal 32,000 connections per node to over… | ||
| Incident Archaeology: Extracting Value from Paperwork and Narratives | Clint Byrum | 19m |
A year-long review of incident artifacts (Jira, PagerDuty, Google Sheets, Jupyter) analyzes about ~130 incidents/year, uses roughly 40 analyst hours/year, and… | ||
| An Organizational Response to Incidents: Designing for Smooth Coordination | Laura Maguire | 46m |
Claim: shifting investment from a lone Incident Commander to the crowd of responders reduces mean time to resolution. Evidence: escalation principle and… | ||
| Building an APM with OpenTelemetry and OpenSource | Goutham Veeramachaneni | 18m |
Shows a working open source APM built by instrumenting a demo "telescope shop" app with auto-instrumentation agents and exporting OTLP to an OpenTelemetry… | ||
| Measuring Real-Life Latency of the Internet: A Netflix Story | Thiara Ortiz | 13m |
Netflix measures per-session round-trip-time distributions using t-digest so it can store and merge summaries at scale instead of raw samples. The CDN serves… | ||
| Founder/CTO Perspectives: The Future of Distributed Tracing | Charity Majors, David Cramer, Maggie Johnson-Pint | 42m |
Compares Honeycomb, Sentry and Stanza approaches and makes concrete claims: arbitrarily wide structured data becomes the canonical source of truth from which… | ||
| Lightning Talks | Lerna Ekmekcioglu, Micha Hernandez van Leuffen, Sal Furino, Nele Lea Uhlemann, Robert Barron, Jonah Horowitz, Prathamesh Sonpatki, Paige Cruz | 36m |
Quick, concrete takeaways: a TLS primer explains use of AES + Diffie‑Hellman key exchange, certificates from CAs and when symmetric vs asymmetric crypto is… | ||
| Human Observability of Incident Response | Matt Davis | 39m |
Uses music-based exercises and weekly practice of practice sessions to build improv skills that equal adaptive capacity; runs a live wheel of expertise game… | ||
| Far from the Shallows: The Value of Deeper Incident Analysis | Courtney Nash | 40m |
A claim: shallow metrics mislead — histograms from over 10,000 public reports in VOID (Verica Open Incident Database) show heavily skewed durations so that… | ||
| How SRE Makes Electric Vehicles | Adam Shake | 35m |
Rivian applies SRE practices to manufacturing by building an automated provisioning and observability platform: PXE/pixie for OS installs, AWX + Ansible for… | ||
| Warding against the Dark Arts: Crafting a Defense Strategy against Botnet DDoS | Shirleen Sharma, Aaron Heady | 42m |
Large-scale DDoS volume rises ~95 percent and monthly traffic from ~100 to ~330 exabytes so the team builds a layered defense: hardened logs and metrics, an… | ||
| The Revolution Will Not Be Terraformed: SRE and the Anarchist Style | Austin Parker | 19m |
Argues that SRE is a socio-technical movement reacting to the commodification of Agile and DevOps, tracing roots to anarchist ideas like mutual aid and Murray… | ||
| Implementing SRE in a Regulated Environment | Sandeep Hooda, Fabian Tay | 20m |
DBS started SRE in 2020 and reports concrete outcomes: a homegrown chaos tool raccoon 60+ attack vectors integrated into CI/CD, capacity predictor Cape… | ||
| Financial Resiliency Engineering: Taming Cloud Costs | Darren Worrall | 38m |
Shopify narrows cloud spend by turning billing into traceable profiles, exporting billing and service data into flame-graph billing profiles (speedscope/prep)… | ||
| Sto: A Better Way to Store and Query Profiler Data | Patrick Somaru | 13m |
Stores sampled stacks as a directed acyclic graph and claims reduces footprint by roughly five orders of magnitude at scale (toy apps see ~2 orders), with an… | ||
| Chaos-Driven Development: TDD for Distributed Systems | Dhishan Amaranath, Tucker Vento | 18m |
Chaos experiments are applied as TDD to a log aggregation HA design on public cloud, integrating alarms, metrics and failover logic before running faults… | ||
| Adaptive Concurrency Control for Mixed Analytical Workloads | Dan Kleiman | 31m |
Klaviyo’s real-time Query Service hits intermittent waves of timeouts despite adequate capacity; investigation shows queuing and variable query latency are the… | ||
| If I Can Do It on an Ambulance, You Can Do It in an Office: Scalable Incident | Thai Wood | 35m |
Adapts the Incident Command System into a small-team, practice-first approach: start as a seed at team level, assign three hats (organizer, connector, expert)… | ||
| How To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations | Vijay Samuel, Nick Pordash | 39m |
Platform ingests ~41M/sec from 1.25M Prometheus endpoints with ~3 billion active series and ~8k queries/sec, up from 2M/sec in 2017. Built a Cortex‑inspired… | ||
| Your Infrastructure Needs to D.I.E. | Timothy Mamo | 31m |
Claims that moving to a distributed immutable ephemeral architecture raises the attacker’s cost, illustrated by a 2003 Antwerp diamond heist that netted about… | ||
| Not All Minutes Are Equal: The Secret behind SLO Adoption Failure | Michael Goins, Troy Koss | 29m |
Claim: minute-by-minute time-slicing hides real reliability signals so time-slicing fails for many services. Using 30‑ and 60‑day traffic and failure logs the… | ||
| Hell Is Other Platforms | Alex Hidalgo, Andrew Clay Shafer | 27m |
Argues that platforms inevitably appear and one-size-fits-all SRE/devops prescriptions fail, citing AWS EC2, the SRE book and NIST service models as concrete… | ||
| nothing matches | ||