Wait for Us! Evolving On-Call as Your Company Grows
summary
Datadog scales from a small startup to ~800 employees ingesting trillions of datapoints per day, and primary/secondary on-call breaks down under that load. Concrete fixes include runbooks in GitHub (markdown + diffable history), monitor feedback links (four Google-form answers attached to every alert), spreadsheets that measure alert volume, flap rates and time-to-recover, Github Issues and Projects for handoffs and training, weekly scheduled Monday handoffs run by a neutral facilitator, shadow ops and scripts showing what changed since last on-call — these reduce noise and spread operational knowledge.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:16Agenda
- 0:49Tricia Weed
- 1:50What is Datadog
- 2:42Early Startup Days
- 3:42Critical Trust
- 4:16Hyper Growth
- 5:37Chasing Years
- 7:29OnCall Teams
- 7:47Managing OnCall Teams
- 9:47Flappy Alerts
- 10:34Hours by Day
- 11:24Trends
- 11:53Alert Data
- 12:28Monitor Feedback
- 14:50Add People
- 16:00Safety
- 16:45Documentation
- 19:11Handoffs
- 21:10Incident Response
- 22:07Conclusion
related talks
26mBuild a data-driven on-call workflow for your team with atomic habits
23mFirefighting: The Engineer's Guide to On-Call
31mHow to Build a Healthy On-call Culture
16mHuman Centered On Call Practices
19mIts Time We Rethink On-Call Culture
44mYou build it, you run it
27mYou Should Be On Call, too
50mYou build it, you run it (why developers should also be on call)