Keynote: Incident Investigation at Planetary Scale
summary
Telemetry-first incident response for a global CDN that handles ~100 TB/s and quarter of a million servers across 1,500 networks and 136 countries uses continuous queries, aggregators and severity-driven escalation to detect and mitigate problems before loss. The homegrown Akamai Query system exposes 11,500 edge tables, pulls ~17 billion rows into network aggregators (mega aggregators see ~5.2B) and runs ~12.7M queries/day (8M alerting); alerts require three consecutive cycles to avoid blips. Focus on phase zero, automated diagnostics, clear severity cards and post-incident learning.
AI-written summary · about
chapters (6)
- 0:00Intro
- 0:25Alert! What's the incident?
- 8:25Incident Severities
- 11:04Akamal Query
- 20:06Hazard Impact Analysis
- 24:30Mitigation Prioritization Analysis
related talks
17mIncident Response @ Scale-Building a Next Generation SOC
17mCloud-Native O11y: Building Scalable Enterprise Systems
27mAccelerating Incident Response With OpenTelemetry and MCP: Building...
44mProduction Incident Evaluation as Cost of Quality
19mHow to handle incidents at scale
30mEiffel: A Tool to Oversee Incident Response From the Heights
46mBig Data Intelligence
25mThe Importance Of Real time Data In Fighting Cyber Attacks