Meatbag Systems: How Our Reliability Culture & Practice...
summary
Zalando had >3,000 registered apps and a messy on‑call landscape; a Feb 2019 bug blocks checkout for ~2,000 customers and exposes noisy alerts, so teams define 21 critical business operations and deploy adaptive paging to page only customer‑impacting incidents. Weekly alerts fall from ~300 to ~20. A May 2022 outage traces to Undertow off‑heap growth; a custom heap‑dumpper and OpenTelemetry‑based observability SDKs pin the root cause and a memory cap fixes it.
AI-written summary · about
chapters (7)
- 0:00Introduction to SRE at Zalando
- 1:26Early days of SRE adoption
- 4:08Building the SRE foundation
- 6:06Learning through incidents
- 11:06Refining incident thresholds
- 13:36Scaling learning and observability
- 18:55Evolution of the SRE department
related talks
51mA Field Guide to Reliability Engineering at Zalando
30mThe Meat of It
18mBeyond the Noise: Effective Management of Non-Fatal Issues
38mEffective App Monitoring in Production
28mOn-Call Experience at Uber
56mGetting all the 99.99(9) you always wanted
22mWhat can possibly go wrong? The story of an outrage at Booking.com
48mWhen things go wrong