Continuous Profiling for GPUs
summary
Continuous profiling extends an existing platform to deliver always-on visibility into GPU behavior by sampling with Linux EVPF and querying NVIDIA NVML; it records stacks 20–100 samples/sec with under ~1% CPU overhead and ~4MB memory, captures per-process and node utilization, power, temperature, PCIe throughput, and now GPU time profiling that measures kernel-level start/stop times so functions show actual GPU-duration in flame charts. Deployment uses a daemonset or binary on Linux and requires no app instrumentation, and early users report actionable hotspots and cost savings.
AI-written summary · about
chapters (7)
- 0:00Introduction to talk
- 0:46What is profiling
- 3:37Why continuous profiling
- 5:43GPU metrics overview
- 7:08CPU-GPU correlation
- 8:11GPU time profiling
- 10:37How to get started
related talks
28mAuto-instrumentation for GPU Performance Using eBPF
27mA Profile-Driven Dynamic Optimization Framework for GPU Applications
29mAuto-instrumentation for GPU performance using eBPF
11mPer-iteration PyTorch Visibility...
28mArray Programming on GPUs: Challenges and Opportunities
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
41mBeyond Uptime: Revolutionizing Fintech Reliability
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads