When things go wrong
summary
Doctolib describes operating one large Rails app for millions of patients and 100,000 practitioners and the range of incidents they face, from calendar glitches to hospital and teleconsultation outages. They built an in-house incident app that hosts checklists, an automated scribe, and postmortems, added pre-deploy DB migration guards and automated cookie checks, and detect problems via business golden metrics plus Slack/SMS alerts. Mitigations include a waiting-room queue, autoscaling adjustments and 2-minute rollbacks; their process evolved from Excel to full automation and yields about one incident per day.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:02First production code
- 2:26Incident management at Dr Libre
- 4:52When to trigger a crisis
- 6:58Red button and false alerts
- 8:02Crisis cell roles
- 10:10Mitigation before fix
- 14:20Post-mortem and lessons
- 20:34Auto-scaling and waiting room
- 27:18Avoiding repeat errors
- 29:26Crisis management evolution
- 33:35English in crisis management
- 35:23Growth and complexity
- 37:19Process evolution
- 41:02Choosing the incident manager
- 42:53Q&A
related talks
48mHow Doctolib Weathered the COVID Crisis: A Final Retrospective
15mA Tour of Doctolib Production
22mWhat can possibly go wrong? The story of an outrage at Booking.com
51mDatabase Incident Management: When Everything Goes Wrong
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures
05mDevOps and incident management: A recipe for success
26mMission Critical Mindset Building a Culture of Resilience in DevOps Teams
20mData Incidents - When things go wrong