AI Observability At Meta Scale
summary
Scalable observability infrastructure at Meta collects host and GPU metrics across 21 data centers using daemons like dynalog and fleet profilers such as strobe light and Kinato, surfaces GPU SM utilization, occupancy and heuristic FLOPS/sec, and stitches CPU–GPU timelines for root cause. Automated analysis flags regressions (operator spikes up to 15× slower), guides actions (removing an unnecessary large layer‑norm yields a 4x QPS gain), and supports 2–3× wins on large‑footprint model optimizations.
AI-written summary · about
chapters (6)
- 0:00AI observability at Meta scale
- 2:12Telemetry and monitoring
- 6:06Performance profiling tools
- 9:53Profiling analysis platform
- 16:10Fleet regression tracking
- 21:52Future directions & Q&A
related talks
29mAI Driven Observability based on Open Source
30mFrom Black Box To Insight: Observability for AI Agents in Production
23mNetwork Communication Debuggability and Observability at Scale
26mAI-Driven Cloud Observability: Reduce toil and amplify time to resolution
01h 00mSystems @Scale Summer Q&A Week 4
29mNon-Invasive AI Agent Observability With OBI
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
40mOptimizing AI/ML Workloads