A Practical Guide To Benchmarking AI and GPU Workloads in Kubernetes
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 11:00· Level 1 | Hall Entrance S10 | Room B·
summary
Step-by-step Kubernetes benchmarking that actually deploys servers, runs load, and returns metrics: uses NVIDIA Triton (HTTP/gRPC), the Triton Performance Analyzer (example: 200 rps, reports p50/p95/p99 and throughput, e.g. 3,600 requests), and a new GI PF mode for generative AI time‑to‑first‑token and inter‑token latency analysis. FMPerf provides a Python API to create cluster, model, and workload specs, automates YAML and pods, and collects GPU stats via nvidia‑smi/GPUstats (seen util ~21–22% in demo) for reproducible comparisons.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:04Triton Inference Server overview
- 5:26Performance Analyzer for workloads
- 7:56Monitoring GPU with Nvidia SMI
- 9:01Live demo: Triton server testing
- 13:15Introducing FMPerf framework
- 18:15FMPerf features and usage
- 24:45Live demo: FMPerf in action
- 28:43Future of AI benchmarking
- 29:04GPU Burn for workloads
- 30:06Summary
related talks
23mBenchmarking Machine Learning Workloads on Kubeflow
27mHarnessing Kubernetes for Scalable AI/ML Workloads
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
38mKubernetes for AI Workloads – What Works and What… Doesn’t
36mProduction GPU Cluster with K8s for AI and DL Workloads
40mPanel: Kubernetes and GPU Trends for AI in Financial Services
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...