Dashboards & Dragons: Crafting SLOs To Tame the AI Platform Cha...
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 11:15· Level 1 | Hall Entrance S10 | Room B·
summary
A multi-cluster AI inference platform is instrumented with Envoy, kserve, OpenTelemetry and Prometheus to turn noisy metrics into actionable SLOs and dashboards; key signals include TTFT 400–700 ms and intermittent ITL spikes up to 1,400 ms, 28‑day success dipping to 99.9%, GPU utilization often below 40%. Burn‑rate alerting is concrete: a 1‑hour burn >13.5 consumes a 30‑day error budget in ~2 hours. Dashboards tie percentiles, token throughput and traces to prioritize fixes and catch slow degradation before user complaints.
AI-written summary · about
chapters (12)
- 0:00Quest for SLO mastery
- 1:04AI platforms at Bloomberg
- 3:30Gen AI platform complexity
- 9:13Dashboards failing teams
- 10:58Crafting SLOs from scratch
- 13:47Dashboards driving action
- 17:43LLM inference latency metrics
- 21:57Inference performance monitoring
- 23:45GPU utilization trends
- 25:32Burn rate alerts for SLOs
- 27:20Fine-tuning SLOs with burn rate
- 29:29Observability for AI platforms
related talks
31mDashboards & Dragons: Reliability Magic for AI Platforms
25mAn SLO-Driven Approach to Enhance Kubernetes Cluster Reliability
40mOptimizing AI Applications for Production with Observability
23mGrafana SLO Demo: Prioritize critical resources with SLO-driven IRM
31mServerless Observability: a case study of SLOs
40mScaling SLOs with Kubernetes and Cloud Native Observability
26mAI-Driven Cloud Observability: Reduce toil and amplify time to resolution
43mSLO Burn—Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size