LISA17
who it's for
site reliability engineers, devops engineers, operations engineers, platform engineers, observability engineers, security engineers, engineering managers
topics
distributed tracing with Zipkin and Jaeger; fast automatic log parsing to Logstash filters; multi-region MySQL architecture; immutable infrastructure and canary deploys; incident response runbooks and orchestration; Linux container performance analysis; BPF-based JVM production monitoring; cloud key management and secret rotation; resiliency testing with Toxiproxy; capacity modeling and Universal Scalability Law; ChatOps and operational bots; distributed firewall and network disaggregation
| title | author | length |
|---|---|---|
| Where's the Kaboom? There Was Supposed to Be an Earth-Shattering Kaboom! | David N. Blank-Edelman | 25m |
Operations teams are awful at retiring systems and should borrow demolition practices: treat removal as reverse engineering, require a written deconstruction… | ||
| Coherent Communications—What We Can Learn from Theoretical Physics | Kevin Barron | 40m |
Evidence shows quantum coherence in algae routes nearly all harvested energy to the nearest chromophore despite receptors being ~2x farther apart than in… | ||
| Persistent SRE Antipatterns: Pitfalls on the Road to Creating a Successful | Blake Bisset, Jonah Horowitz | 47m |
Argues that alerts must target user-level service health not individual hosts, and that noisy host alerts cause alert fatigue; recommends automated remediation… | ||
| The 7 Deadly Sins of Documentation | Chastity Blackwell | 43m |
Seven concrete failures and fixes: documentation ends up deprioritized so incidents drag on, and a GitHub 2017 survey finds 93% insufficient; separate run… | ||
| Working with DBAs in a DevOps World | Silvia Botros | 30m |
Advice: put a technology catalog in place listing every cache, datastore and framework and require a vetting process before adding things like Cassandra or a… | ||
| Case Study: Deploying a Multi-Region, Highly Available MySQL Architecture | Gabriel Ciciliani | 41m |
Designs a three-region Azure MySQL deployment with a single-master (chosen for lowest latency) and local read replicas in each region, traffic routed by Azure… | ||
| Distributed Tracing: From Theory to Practice | Stella Cotton | 30m |
Heroku describes running an in-house Zipkin deployment (Java app, Docker/Cassandra) and instrumenting Ruby services with rack middleware and Faraday hooks to… | ||
| Stories from the Trenches of Government Technology | Matt Cutts, Raquel Romano | 33m |
Stories describe building the vets.gov platform with a thin Rails API and React front end that cuts processing from 100+ days to one day, raises online… | ||
| Fast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs | Biplob Debnath, Will Dennis | 30m |
Automatic parser called LogMine clusters tokenized logs, masks datatypes, aligns variants with Smith-Waterman and outputs Logstash filters that integrate with… | ||
| Failure Happens: Improving Incident Response in Large-Scale Organizations | Damon Edwards | 45m |
Describes concrete fixes for slow enterprise incident response: turn manual, ticket-driven handoffs into operations as a service with a secure hub… | ||
| Opening Plenaries: Security in Automation | Jamesha Fisher, Leigh Honeywell | 38m |
Argues that automation must handle the repetitive security work that scales: scripted user onboarding/offboarding via Slack command-based board bot, Duo-driven… | ||
| Queueing Theory in Practice: Performance Modeling for the Working Engineer | Eben Freeman | 46m |
Small experiments measure one core’s latency under load (thousands of req/s, sub-millisecond latencies) and fit a simple M/M/1-style model with a… | ||
| Clarifying Zero Trust: The Model, the Philosophy, the Ethos | Evan Gilman, Doug Barth | 40m |
PagerDuty documents building a cross‑region access control system that replaces VPN tunnels with a full opportunistic IPSec mesh so mutual auth and… | ||
| Fast and Safe Production Monitoring of JVM Applications with BPF Magic | Sasha Goldshtein | 45m |
BPF + BCC tools attach to JVM static tracepoints and kernel events and produce aggregated results in-kernel instead of dumping massive perf files. Examples… | ||
| Linux Container Performance Analysis | Brendan Gregg | 43m |
Describes Netflix Titus as a runtime that launches 1,000,000 containers in one week and uses Linux 4.9+, Intel snap and Netflix Vector/Atlas for metrics. Shows… | ||
| SREBot—More Than a Chatbot—An Intelligent Bot to Crush Mitigation Time | Cezar Alevatto Guimaraes | 32m |
An intelligent incident bot called SREBot automates bridge creation, chat logging and ticketing, uses LUIS for NLP to parse commands, and runs automated impact… | ||
| Sample Your Traffic but Keep the Good Stuff! | Ben Hartshorne | 30m |
Describes concrete sampling recipes: record per-event sample rates, use fields like HTTP status, dataset, method and URL to bias selection, and let servers… | ||
| Wait for Us! Evolving On-Call as Your Company Grows | Christopher Hoey | 24m |
Datadog scales from a small startup to ~800 employees ingesting trillions of datapoints per day, and primary/secondary on-call breaks down under that load… | ||
| An Internet of Governments: How Policymakers Became Interested in “Cyber” | Maarten Van Horenbeeck | 45m |
Covers concrete episodes that pushed governments into cyber policy: Clipper Chip (1993), Stuxnet (2010), DigiNotar (2011) and the Mirai DDoS that toppled DNS… | ||
| Charliecloud: Unprivileged Containers for User-Defined Software Stacks in HPC | Michael Jennings | 45m |
Charliecloud uses Linux user and mount namespaces to run industry-standard Docker images on HPC without privileged daemons, claiming a minimal trusted runtime… | ||
| Pintrace: A Distributed Tracing Pipeline | Suman Karumuri | 46m |
Pinterest builds a Zipkin-based pipeline called PinTrace that collects spans via local logs → Kafka (using singer) → a Spark job into Elasticsearch, handling… | ||
| System Crash, Plane Crash: Lessons from Commercial Aviation and Other Engineering Fields | Jon Kuroda | 01h 15m |
Comparisons use concrete incidents and fixes: Air France 447 stalls and Air France 447 228 dead expose sensor failure plus missing manual training, United 232… | ||
| DevOps in Regulatory Spaces: It's Only 25% What You Thought It Was… | Peter Lega | 44m |
DevOps teams scale into regulated health‑science by turning tools into auditable platforms: a Greenfield DevOps stack evolves into a qualified Robot Framework… | ||
| Testing Before You Scale & Making Friends While You Do It | Renee Lung | 27m |
Builds a Node-based load generation platform to simulate one account creating up to 20,000 incidents/day (≈7.2M/year) and runs three production nuclear option… | ||
| UX Design and Education for Effective Monitoring Tools | Amy Nguyen | 37m |
Argues that UX changes measurably improve monitoring: adds an alert previewer that shows aggregation (average over 10 minutes) and notification windows so a… | ||
| ChatOps at Shopify: Inviting Bots in Our Day-to-Day Operations | Daniella Niyonkuru | 46m |
Shopify runs chat-driven ops via a Ruby Lita-based bot called Spy tied to Slack, GitHub, PagerDuty and internal APIs, serving 500k+ merchants and ~1,200… | ||
| Your Secrets in Cloud-Based Key Management Services | Dan O'Boyle | 28m |
Stack Overflow moves secrets from repos into Google KMS, storing ciphertext in a Google Cloud Storage bucket and decrypting at runtime with a custom kms.py… | ||
| Vax to K8s: Ticketmaster's Transformation to Cloud Native Devops | Heather Osborn | 44m |
Ticketmaster moves 200+ products from a VAX/on‑prem estate into a cloud‑native DevOps model using Docker, Kubernetes (Tectonic/CoreOS), AWS and Terraform… | ||
| Resiliency Testing with Toxiproxy | Jake Pittis | 26m |
Shopify builds and runs an open‑source TCP proxy called Toxiproxy that sits between apps and dependencies, exposes an HTTP API, and has Ruby and Go clients so… | ||
| Capacity and Stability Patterns | Brian Pitts | 41m |
Eventbrite handles sudden on‑sales that can quadruple one endpoint’s traffic in a minute and serves 150M tickets last year, so engineering focuses on concrete… | ||
| Never Events | Matt Provost | 46m |
Adopts the NHS definition of a Never Event (4 criteria: wholly preventable, serious harm potential, past occurrence, easily recognized) and shows how to make… | ||
| "Don't You Know Who I Am?!" The Danger of Celebrity in Tech | Corey Quinn | 45m |
Talk argues that copying famous companies without context fails; cites Netflix has developers with root in production and explains why that works only with… | ||
| Scaling Talent: Attracting and Retaining a Diverse Workforce | Derek Arnold, Amy Nguyen, Qianna Patterson, Wayne Sutton, Derek Watford, Tameika Reed | 01h 37m |
Panelists argue that fixes are concrete not rhetorical: companies build internal technical‑education platforms, create paid high‑school internships, recruit… | ||
| Have You Tried Turning It Off and Turning It On Again? | Tanya Reilly | 40m |
Argues that a simultaneous reboot can leave a site unrecoverable because of hidden runtime dependencies and bootstrapping cycles, shows dependency graphs… | ||
| Disaggregating the Network: Switching as a Service | Nina Schiff | 37m |
Facebook disaggregates switching hardware and software by building white‑box switches (Wedge, Backpack) that run FBOSS on a CentOS microserver attached to a… | ||
| Scalability Is Quantifiable: The Universal Scalability Law | Baron Schwartz | 30m |
Claim: scalability is a simple quantitative function. The Universal Scalability Law models throughput vs concurrency with two measurable parameters: sigma… | ||
| Now You See Me Too: Visual Tooling for Advanced System Analysis | Suchakrapani Sharma | 44m |
Shows how large kernel and user traces compress into actionable visuals using FlameGraphs, FlameCharts and timelines, and demonstrates concrete tooling… | ||
| Managing SSH Access without Managing SSH Keys | Niall Sheridan | 28m |
Replaces long-lived private keys with ephemeral certificates signed by a central casher CA that authenticates via Google OAuth, issues keys generated in memory… | ||
| Calcifying Crisis Readiness | Rock Stevens | 20m |
Argues organizations must run no-notice full-scope penetration tests in production, combine Chaos Monkey-style fault injection with military “react to contact”… | ||
| Debugging at Scale Using Elastic and Machine Learning | Mohit Suley | 30m |
A production pipeline ingests Kafka/Storm traces and user impressions, runs distributed tracing via Zipkin, tokenizes logs into sentences and key/value pairs… | ||
| LinkedIn's Distributed Firewall | Mike Svoboda, Nils Christian Roscher-Nielsen | 41m |
LinkedIn shifts firewall enforcement from big‑iron choke points to per‑host rules that are generated from a central ACL DB and applied via iptables and ipset… | ||
| Operational Compliance: From Requirements to Reality | Trevor Vaughan | 26m |
Onyx Point operationalizes compliance by integrating scanners and policy checks into GitLab CI pipelines, using open tools like the SCAP Security Guide… | ||
| The Actor Model and the Queue or “Batch is the New Black” | James Whitehead II | 46m |
Combines the Actor model with queue-based pull not push processing and shows it runs at national scale: re-architects an enrollment pipeline that must handle… | ||
| The Hidden Costs of On-Call: False Alarms | Cody Wilbourn | 32m |
False alerts multiply costs: simple investigation time plus a ~40% context‑switch penalty produces roughly a 4× overhead and a 5% false‑alert rate can cost… | ||
| Becoming a Plumber: Building Deployment Pipelines | Daniel Barker | 45m |
Claims a company-wide system that enforces deployment pipelines for every app and lets many app types use the same pipeline structure, cutting legacy drift and… | ||
| nothing matches | ||