SREcon16 Europe
2016-07-11..2016-07-13
Dublin, Ireland
| title | author | length |
|---|---|---|
| Splicing SRE DNA Sequences in the Biggest Software Company on the Planet | Greg Veith | 44m |
Azure creates an SRE org from scratch across ~58 service offerings and >200 company services, hires a small core team (~6–7 pioneers from Google/Amazon/etc.)… | ||
| Doorman: Global Distributed Client Side Rate Limiting | Jos Visser | 32m |
Doorman is a Google system that lets globally distributed clients coordinate so total requests never exceed a shared capacity; it is implemented as a tree of… | ||
| Flash Sale Engineering | Emil Stolarsky | 19m |
Engineered a queuing layer in the load balancer using Nginx + OpenResty Lua, a per-LB leaky bucket that only admits a calculated share of global checkout… | ||
| Managing Up and Sideways as an SRE | Liz Fong Jones | 24m |
Describes concrete tactics SREs can use without title change: mentorship as mentoring, coaching, sponsoring, using Situation-Behavior-Impact feedback, and… | ||
| The Production Engineering Lifecycle: How We Build, Run, and Disband Great... | Andrew Ryan | 36m |
Facebook describes a deliberate lifecycle: hire production engineers as strong coders with systems experience and vet them in software-engineer coding… | ||
| How to Improve Your Service by Roasting It | Jake Welch | 17m |
A practical method for surfacing hidden operational debt: run a series of short, interactive deep dives across the service (architecture diagrams… | ||
| What SRE Means in a Start-up | Brian Scanlon | 21m |
Describes how SRE at Intercom runs on AWS with ELB, nginx and a monolithic Ruby on Rails app, with about 60 engineers in Dublin, owning the build and deploy… | ||
| Capacity Planning at Scale | Ramón Medrano Llamas | 31m |
SREcon16 Europe - Capacity Planning at Scale
Ramón Medrano Llamas, Google
Have you ever bought machines? What if you need to even build datacenters? How can… | ||
| Load Shedding—Approaches, Principles, Experiences, and Impact in Service Management | Acacio Cruz | 47m |
Google-style load shedding deploys a local, fast verdict system that uses in‑flight per‑request CPU accounting, soft/hard thresholds and tagged traffic to… | ||
| Tier1 Metamorphoses | Nina Mushiana | 29m |
LinkedIn automates known alerts and repurposes NOC time to train engineers, cutting a NOC zone’s alert load from ~35,000 to 5,000 and automating ~37,000… | ||
| Incident Response @ FB, Facebook's SEV Process | Gareth Eason | 39m |
Explains Facebook’s incident workflow: on‑call discovers incidents, escalates to an imoc who coordinates (not fixes), uses IRC logs, Messenger and a… | ||
| Production Improvement Review: Taking a Bite Out of Repair Debt | Martin Check | 38m |
Azure SRE runs a Production Improvement Review that forces common KPIs across wildly different teams, tracks time-to-detect/engage/fix, and pushes incident… | ||
| The Many Ways Your Monitoring Is Lying to You | Sebastian Kirsch | 39m |
Monitoring systems misrepresent reality: missing points produce fake spikes when deltas are computed, sampling granularity hides short bursts, and automatic… | ||
| Next-generation Alerting and Fault Detection | Dieter Plaetinck | 40m |
Machine learning and stream processing often underdeliver for alerting: commercial attempts (Etsy Skyline) and large-scale ops favor simple, maintainable… | ||
| The Next Linux Superpower: eBPF Primer | Sasha Goldshtein | 38m |
Kernel-side programmable tracing with eBPF plus the bcc front end compiles C into safe bytecode, JITs it, and attaches to kprobes, uprobes and tracepoints… | ||
| The Virtuous Cycle: Getting Good Things out of Bad Failures | Joy Scharmen | 35m |
Heroku runs over a million Postgres databases, tens of thousands of Redis instances and hundreds of thousands of dynos and the same integer auto‑increment… | ||
| The Structure and Interpretation of Graphs | Niall Murphy | 38m |
Shows concrete graph failures and fixes: latency percentiles (50/85/95/99) on a log scale reveal huge tail impacts and timeouts; QPS hourly spikes expose… | ||
| Alerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise | Björn Rabenstein | 31m |
Company runs ~100 engineers with a ~5 person production engineering team and replaces noisy host checks with symptom-based alerts so pages reflect user impact… | ||
| Challenges of Machine Learning at Scale | Graham Poulter | 41m |
A practical design for an ad-click pipeline that makes 100,000 queries/sec, handles gigabytes/sec of logs and trains over billions of features uses map-reduce… | ||
| Panel: Oncall | Laura Nolan | 42m |
Panel discussion shows concrete practices: mandatory shadow and reverse-shadow onboarding, automated paging into rotations, tracking metrics like first-touch… | ||
| Lessons from Automatic Incident Resolution for a Million Databases | Greg Burek | 22m |
Heroku runs about one million databases across tens of thousands of AWS instances and reduces pages by monitoring the single business metric — whether a… | ||
| Moving a Large Workload from a Public Cloud to an OpenStack Private Cloud... | Nicolas Brousse | 22m |
Company migrates high-volume, low-latency bidding and caching workloads off public cloud into OpenStack on Ubuntu, automates bare-metal provisioning and cloud… | ||
| My Service Runs at 99.999%...All Those Tweets about Outages Are Not Real... | Kumar Srinivasamurthy | 22m |
Claims that a service runs at 99.999% while users hit errors are dissected with concrete failures: a Cortana UI shows an empty canvas not logged as failure… | ||
| Availability Objectives of SoundCloud’s Microservices | Bora Tunca | 16m |
SoundCloud sets a minimum availability for each microservice and audits all ~140 services to find risky ones. Availability uses two thresholds: error rate and… | ||
| Downtime Budgets | Cory Lueninghoener | 29m |
Adapts the error‑budget idea to HPC by tracking per‑cluster downtime budget (one 10‑hour DST per month → 30 hours per quarter ≈ 1.4% time) across ~20 clusters… | ||
| Relieving Technical Debt through Short Projects | Liz Fong Jones | 21m |
Teams run one- to two-week interrupt reduction projects on a weekly rotation so one site handles most pages while the other focuses on short automation, docs… | ||
| Running Storage at Facebook | Federico Piccinini | 21m |
Facebook moves blob storage from HDFS to a Warm Storage system that supports mixed encodings and a draining helper service that picks which hosts to evacuate… | ||
| Linux Kernel Building, Testing, and Deployment at Facebook | Yannick Brosseau, Phillip Duncan | 36m |
Facebook moves to an upstream first kernel workflow, building kernels in containers that produce reproducible RPMs on an hourly cadence for every diff, plus… | ||
| Past, Present, and Future of Network Operations | David Barroso | 18m |
Argument that networking finally adopts automation, with concrete parts: modern boxes expose Linux/open APIs and streaming telemetry replaces SNMP… | ||
| Bridging Multicast to the Cloud | Daniel Emord | 14m |
A legacy Windows app relies on multicast for state sync and cloud providers block multicast, so the team implements an overlay using n2n mesh VPN to carry… | ||
| Full-Mesh IPsec Network: 10 Dos and 500 Don'ts | Fran Garcia | 33m |
Deploys a full mesh IPsec network across hundreds of hosts using ipsec-tools and raccoon, achieves ~800 Mbps on a 1 Gbit link, and completes a three month… | ||
| Extreme OS Kernel Testing | Kirk Russell | 28m |
A fuzz framework captures real kernel calls and filesystem objects from DTrace then randomizes them as a mad‑lib ordering; applied to FreeBSD it triggers a… | ||
| DNS: Old Solution for Modern Problems | Thomas Jackson, Rauf Guliyev | 29m |
LinkedIn implements a distributed, highly available DNS-based service discovery written in Go that fronts multiple topology backends via plugins, uses… | ||
| Scaling Shopify's Multi-Tenant Architecture across Multiple Datacenters | Florian Weingarten | 39m |
Shopify moves from a single big MySQL to per-shop database sharding, splitting ~275,000 merchants so each shop’s rows live in one primary with cross-datacenter… | ||
| Leading a Team with Values | Rich Archbold | 27m |
Introduces four concrete team values—security availability performance scalability cost, faster safer easier shipping, zero‑touch ops and run less software—and… | ||
| The Knowledge: Towards a Culture of Engineering Documentation | Riona MacNamara | 40m |
Google finds 48% of engineers cite missing trustworthy docs and discovers 137,000 internal Sites with 90% unused, so engineers embed markdown docs next to code… | ||
| Bridging the Safety Gap from Scripts to Full Auto-Remediation | David Mah | 20m |
Dropbox builds an alert → diagnose → authorize → remediate pipeline: diagnose plugins re-check alerts every five minutes and emit a concrete prescription… | ||
| Active Fault Finding in Networks | Richard Sheehan | 32m |
Describes building active probes at Facebook scale: open-sourced fbtracer to enumerate ECMP paths, a UDP pinger/responder system called net-norad with ~1:100… | ||
| Fixing the Internet for Real-Time Applications (Games) | Adam Comerford | 34m |
Riot builds its own global backbone called Riot Direct, buys fiber, deploys PoPs (Amsterdam, Warsaw, mini-PoP in Poland) and pays for paid peering with large… | ||
| Data Privacy Legislation and the Impact on SRE | John P. Looney, Simon McGarr | 45m |
Court of Justice invalidates the EU–US Safe Harbor, halting its transfers immediately and forcing engineering teams to choose between Standard Contractual… | ||
| Techniques and Tools for a Coherent Discussion about Performance in Complex... | Theo Schlossnagle | 38m |
Argues that latency comes from the spaces between services and recommends two concrete practices: distributed tracing (using zipkin/dapper tracing with sampled… | ||
| Government Needs SRE | Mikey Dickerson | 53m |
After rescuing HealthCare.gov in 2013 this group forms the U.S. Digital Service in 2014 and builds concrete wins: a scalable login system rewritten for about… | ||
| Securing RFIDs by Randomizing the Modulation and Channel | Haitham Hassanieh, Jue Wang, Dina Katabi, Tadayoshi Kohno | 21m |
RF‑Cloak claims to protect deployed commercial RFID cards without modifying them by randomizing modulation and the channel; it provides theoretical proofs and… | ||
| nothing matches | ||