SREcon24 Europe/Middle East/Africa
2024-10-29..2024-10-31
Dublin, Ireland
who it's for
site reliability engineers, platform engineers, production engineers, observability engineers, devops engineers, security engineers, database operators
topics
service level objective thresholds; OpenTelemetry tracing at scale; GitHub Actions OIDC workload identity; eBPF schedulers and instrumentation; time synchronization and PTP; MySQL metadata lock behavior; noisy pager and alert reduction; synthetic monitoring and end-to-end testing; large-scale load testing; logs management and reindexing; software supply chain risk management; NVMe over TCP storage performance
| title | author | length |
|---|---|---|
| Dude, You Forgot the Feedback: How Your Open Loop Control... | Laura de Vesine | 36m |
Production control planes routinely accept intent or fire‑and‑forget commands that require operators to go look elsewhere to verify results, causing outages… | ||
| You Depend on Time, This Is How It Works and You Won’t... | Philip Rowlands | 32m |
History of calendars to atomic clocks with concrete numbers and systems: the day is 23h56m4s sidereal, Gregorian skipped Oct 4→15 and Sweden once had Feb 30… | ||
| SRE Saga: The Song of Heroes and Villains | Daria Barteneva | 38m |
Maps the Dungeons and Dragons metaphor onto concrete SRE practices: outlines five team patterns (embedded, platform, short‑term, ongoing capability, central)… | ||
| The Frontiers of Reliability Engineering | Heinrich Hartmann | 39m |
Zalando runs ~3,500 engineers and more microservices than people, deploys on AWS + Kubernetes, uses Docker, Prometheus/Grafana and OpenTelemetry, and claims… | ||
| I Can OIDC You Clearly Now: How We Made Static Credentials a... | Iain Lane, Dimitris Sotirakis | 29m |
Grafana Labs replaces long‑lived static credentials with GitHub Actions OIDC to get short‑lived cloud tokens, using workload identity federation for GCP and an… | ||
| OMG WTF SSO: A Beginner’s Guide to Single Sign-On... | Adina Bogert-O'Brien | 30m |
SSO often feels safer but can hide bigger failures: vendors may leave username/password backdoors, offer SSO as optional, or use a brittle attribute mapping… | ||
| Sailing the Database Seas: Applying SRE Principles at Scale | Ioannis Androulidakis, Martin Alderete | 34m |
Booking.com runs MySQL across bare metal, private and public clouds with ~hundreds of clusters and >10k servers managed by ~20 SRE DBAs, and measures read… | ||
| Survivor: MySQL Island – Outwit, Outplay, Outlast Metadata... | Julia Jablonska | 21m |
Migration from MySQL 5.7 to 8.0 exposes a change: creating a child table with a foreign key requests an exclusive MDL on the parent that can block… | ||
| Fixing Your Noisy Pager in 500 Easy Steps | Chris Sinjakli | 18m |
Pager load drops from hundreds of pages per month to low tens per week by combining a weekly triage loop — group alerts by name, sort by frequency, manually… | ||
| Achieving Excellence: SLO Thresholds That Transform Service... | Thiara Ortiz | 22m |
Netflix Open Connect serves 230 million members from over 18,000 OCAs and measures QoE with three concrete SLIs: play delay, network rebuffer counts, and… | ||
| Selective Reliability Engineering: There Is No Single Source... | Elise Burke | 14m |
Builds an image-upload service and compares two concrete architectures: a single-server with local filesystem versus a stateless service + cloud DB/blob… | ||
| Why You’re (Probably) Doing Service Catalogs Wrong | Lisa Karlin Curtis | 19m |
Service catalogs often take months or years to ship and then languish because teams don’t have the data or the incentive to keep it fresh—efforts spun up a… | ||
| Exploring the Unintended Consequences of Automation in Software | Courtney Nash | 31m |
Research mines the VOID dataset of 10,000+ public incident reports, filters ~450 cases to ~200 for close thematic analysis, and finds automation often… | ||
| Rock around the Clock (Synchronization): Improve Performance... | Lerna Ekmekcioglu | 16m |
Demo shows a Kubernetes e‑commerce site whose tail latency jumps from ~140 ms to ~5 s when a data “elephant” workload shares the link, with system delays… | ||
| Mnemonic Rules for Eponymous Laws or: There’s a Law for That! | Peter Burkholder | 17m |
Walks through ten named laws relevant to SREs and gives concrete mnemonics, examples and outcomes: a cloud platform assembled from every tech stack aims for… | ||
| SRE Stakeholders: A Spotter’s Guide | Dave O'Connor | 31m |
Conference attendance data shows 550 people from ~200 companies with the most common company sending one attendee, and activist investors and churn making… | ||
| Panel Discussion: Is Reliability a Luxury Good? | Emil Stolarsky, Andrew Ellam, Niall Murphy, Joan O'Callaghan, Avleen Vig | 45m |
Companies are cutting reliability teams after rounds of layoffs and treating upkeep as expendable, citing short‑term savings while outages still cost real… | ||
| Enhancing Elasticsearch Performance: Innovative Reindexing... | Leila Vayghan | 28m |
Shopify separates heavy reindex work onto a dedicated reindex node pool on GKE and drives autoscaling with KEDA with a Prometheus trigger that measures shards… | ||
| Lessons from Unix History | Diomidis Spinellis | 34m |
Unix history is reconstructed from GitHub snapshots and a PDP emulator that boots original code, revealing concrete lessons: the original kernel is <2600 lines… | ||
| Treat Your Code as a Crime Scene | Adam Tornhill | 40m |
Argues that version‑control data plus behavioral analysis (the CodeScene tool) reveal where to spend effort: hotspots are typically 1–3% of code but drive most… | ||
| Finding the Capacity to Grieve Once More | Alexandros Kosiaris | 34m |
Wikipedia suffers a major outage in 2020 when a celebrity death converges with a DDoS, exposing parse-thundering and cache invalidation bugs; fixes include… | ||
| Incident Groundhog Day | Hamed Silatani | 30m |
A staged‑world exercise recreates a Black Friday e‑commerce outage triggered by a secret API key rotation plus a cache refresh; 20 incident responders each… | ||
| Anomaly Detection in Time Series from Scratch Using... | Ivan Shubin | 34m |
Claims that simple statistics (not ML) catch slow burns and spikes using Z‑score and MAD; implements it against Graphite, visualizes and alerts in Grafana, and… | ||
| Generative AI: Beyond (Just) Hype | Todd Underwood | 37m |
Hype is exaggerated but not pointless: GPT‑4, Claude, Gemini and Dolly expose real strengths and clear failures for production engineering. Autoscaling works… | ||
| From PIDs to Pods: The Life Cycle of an eBPF-Autoinstrumented.. | Marc Tudurí | 24m |
Automatic instrumentation that attaches eBPF probes to live processes and exports Prometheus or OpenTelemetry metrics and traces, implemented as Grafana Beyla… | ||
| Scheduling at Scale: eBPF Schedulers with Sched_ext | Daniel Hodges | 30m |
eBPF plugs into the kernel via skex ops to run custom schedulers that swap dispatch queues, set vtime slices, and call Kfuncs; a working simple example is… | ||
| When Your SaaS Provider Goes out of Business – Lessons from... | Raphael Seebacher, Christof Gerber | 35m |
On 2 Feb 2023 a critical URL‑filter vendor folds, risking service for 100 customers, 300k daily users across 3,500 sites; response splits into three parallel… | ||
| Configuration Languages Are the Bane of Our Existence | Paul Komkoff | 21m |
Configuration often mutates from simple key=value files into layered DSLs that are untyped, untested programs; examples include sendmail.cf built atop M4… | ||
| Just Buy the Printer: Resilience in Action | Cail Young | 12m |
Four days before a release, a code signing certificate is due to expire and CA/Browser Forum rules have changed how private keys must be stored, forcing two… | ||
| Noisy Neighbors, through Networking | René Treffer, Ben Kochie | 30m |
Reddit-scale clusters (500k–1M requests/sec) show concrete failure modes: one monitoring path generates 140,000 UDP packets per second from four senders… | ||
| Taming Noisy Benchmark Results Using Change Point Detection | Matt Fleming | 16m |
Change Point Detection identifies stable shifts in noisy benchmark time series so you can separate real regressions from random spikes, requires roughly needs… | ||
| Enabling Product Scalability through Load Testing | Monica Baluna, Ehab Tawfik | 18m |
Load testing validates migration of legacy group chats into Forums to support 10x group size, using scripts to create rooms and add users, Grafana dashboards… | ||
| NVMe/TCP Makes iSCSI Look like Fortran | Chris Engelbert | 33m |
Claim: replacing iSCSI with NVMe/TCP nearly doubles read/write IOPS and cuts per‑packet overhead by ~25%, using standard Linux kernel NVMe stack over commodity… | ||
| The Silent Performance Killers: BIOS and Firmware Updates | Darin E. Langone | 17m |
Vendors push BIOS and firmware patches that can silently reduce throughput and increase latency unless you run pre- and post-update benchmarks. Uses redfish… | ||
| How a Single API Endpoint Saved Us 3000 CPU | Lasse Canth Hels | 15m |
Grafana Mimir ingestors on Azure spot nodes suddenly double their held series during rollouts because consistent-hash ownership moves when pods leave the ring… | ||
| Managing the Risk of Software Supply Chain Attacks | Mark P Hahn | 41m |
Ninety percent of companies rely on open source and supply-chain compromises grow explosively (uploader cites a 742% annual rise), so focus on concrete… | ||
| When SRE and Security Teams Meet to Face a Crisis | JR Aquino | 39m |
SRE and security teams trade hard specifics: an IDS update that redirects alerts to root floods an ISP with Port 25 SMTP traffic and causes self‑DOS, Juniper… | ||
| How to Host a (Very) Popular Website for 30 Altairian... | James Beal | 35m |
Archive of Our Own serves about 34 billion pages a year (~87–100M pageviews/day) on owned bare metal to avoid deplatforming, runs on Rails with MySQL/MariaDB… | ||
| How Snowflake Migrated All Alerts and Dashboards to a... | Carlos Mendizabal | 32m |
Migrated all alerts and dashboards to a Prometheus-based metrics system in three months, rewriting ~1,200 unique alerts and 200 dashboards across 70+ teams… | ||
| What If We Ask Linux to Do Cryptography for Us? | Oxana Kharitonova | 28m |
Proposal: move private keys from user space into the Linux kernel and call crypto there from Golang or Rust via system calls, replacing user-space libraries… | ||
| Synthetic Monitoring and E2E Testing: 2 Sides of the Same Coin | Carly Richmond | 32m |
Argues that the same code artifact validates user journeys in CI and production, demonstrating Playwright + Elastic Synthetics run from GitHub Actions to… | ||
| Lightning Talks | Ian Duffy, Roberto Frenna, Konstantinos Fardelas, John Lunney, Dawid Nowak, John P. Looney, Jennifer Petoff, Sal Furino, Robert Barron | 34m |
Nine four-minute briefs give concrete takeaways: an argument for measuring the meantime to recovery of engineers and scheduling to prevent burnout; GCE rollout… | ||
| Monitoring Systems as a Service – Walking the Line between... | Joan O'Callaghan | 33m |
Case study of migrating Udemy off unconstrained Datadog/Honeycomb usage: shows building an observability pipeline to rate‑limit dev logs, splitting retention… | ||
| An Exploration in Storing Telemetry in Cloud Object Storage | Mike Heffner, Ray Jenkins | 36m |
Shows building a telemetry data lake using NDJSON, Parquet, DuckDB and Apache Iceberg with S3 object storage and a small Go CLI to ingest and benchmark… | ||
| Opening the Box: Diagnosing Operating-System Task-Scheduler... | Julia Lawall | 35m |
Linux scheduler decisions cause measurable slowdowns on large multicore runs; visualization and extra trace points reveal schedgraph tools (datograph… | ||
| Embrace Fleet Reboots and Make Them Boring | Everton Didone Foscarini | 35m |
Cloudflare standardizes monthly reboots across 300 edge cities by building an automation stack: Quicksilver for coordinated state, a Bing reboot server… | ||
| A Brief History of Release Engineering | Dinah McNutt | 17m |
History traces release engineering from punch cards and the command line to canaries and CI/CD pipelines; major claim is that tooling changed practice but not… | ||
| Red Tide Revert | David Newman | 18m |
Automated system ingests production logs via an ELK-style backend, correlates recent commits and stack traces, and proposes reverts; service components are a… | ||
| Riot Games: Evolution of Observability at the Gaming Company | Erick Moreira, Kirill Mikhailov | 26m |
Riot replaces scattered homegrown collectors with Vector and cluster agents, injects telemetry via a dynamic admission webhook, and enforces sampling and… | ||
| A Powerful Logs Management Solution We All Have and Use but... | Costa Tsaousis | 20m |
systemd-journal is presented as an underestimated, production-ready logs engine that indexes key=value fields, supports forward‑secure sealing for tamper… | ||
| Blast Radius Reduction for Large-Scale Distributed Systems | Linhua Tang | 16m |
Argues that splitting a global cluster into ~100+ cell-based architecture units limits spatial blast radius but raises infra cost (example: one 100-cell design… | ||
| AppStack: An Open Source Cloud Native Platform for Running... | Dimitris Mitropoulos, Alex Kiousis | 22m |
AppStack is a self-hosted cloud-native platform running on self-hosted Kubernetes with deployments via Helm and Harbor, hosting services used by more than 8… | ||
| Science Reliability Engineering for High Performance Computing | Nicholas Jones | 31m |
Claims an SRE-driven stack called OpenCHAMI replaces hands‑on vendor pets with a containerized, API‑driven boot and lifecycle system built in Go, using Podman… | ||
| Get Your Non-SREs Oncall Ready! | JC van Winkel, Brad Lipinski | 30m |
A self-service training system recreates a deliberately broken photo upload microservice (front end, back end, DB, multi‑datacenter) and a… | ||
| Transforming Production Readiness | Panagiotis Moustafellos | 38m |
Elastic builds an SLO observability product that monitors around 900,000 SLOs across 180+ clusters and 60 regions, ingesting petabytes and ~500 TB/day, using… | ||
| Energy Consumption of Datacenters | Thomas Fricke | 37m |
Claims that hyperscalers need 5 GW per hyperscaler and that the US would need about 150 small modular reactors (300 MW each) (none built domestically) underpin… | ||
| Are We Really Engineers? | Hillel Wayne | 37m |
Interviews with 17 crossovers (12 hours total) probe whether software counts as engineering and report 15 of 17 say yes, with nuance: software’s key… | ||
| nothing matches | ||