Auto-instrumentation for GPU Performance Using eBPF
summary
Grafana’s eBPF-based implementation auto-instruments CUDA driver calls to emit Prometheus metrics for kernel launches per second, grid/block dimensions, and memory copies without app changes. The probes capture cudaLaunchKernel, cudaMemcpy, and malloc events and expose them cluster-wide; fusing kernels raised GPU utilization from ~60% to 100%, cut latency from 48 ms to 19 ms, and reduced hundreds of kernel launches to far fewer while grid size rose ~900→3000. Implementation is open source on GitHub, has limited kernel execution timing and no AMD probes yet.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:46GPU deployment challenges
- 5:19GPU monitoring limits
- 8:08Software-GPU interaction
- 9:52eBPF auto-instrumentation
- 11:15CUDA probes and metrics
- 13:44CUDA memory probes
- 15:32Fused kernels impact
- 17:19Demo: GPU utilization
- 20:53Future of GPU profiling
- 21:55Limitations and tools
- 22:39AMD GPUs and context tracking
- 23:21AI infrastructure observability
- 25:10Q&A
related talks
29mAuto-instrumentation for GPU performance using eBPF
21mAuto-Instrumentation for GPU Performance using eBPF
33mAuto-instrumentation of Prometheus for RED Monitorin...
42meBPF-Powered Observab...
29mUsing eBPF for Non-invasive, Performant, Instant Network Monitoring
35mSidecarless Mesh Monitoring with eBPF and Grafana
18mFind out what your processes are capable of with ebpf
36mPrometheus as exposition format for eBPF programs ..