Understanding and Profiling NVMe-over-TCP Using ntprof
summary
ntprof is a kernel + user tool that models NVMe‑over‑TCP as nine software switches and adds tracepoints to Linux 5.15.143 to collect lossless per‑IO timelines. It reports end‑to‑end and per‑stage latencies, identifies queueing in the TCP/IP stack, and localizes SW/HW bottlenecks. Example: a 4KB random read jumps from 65 μs to 82 μs on the target but from 40 μs to 126 μs in the network when depth increases 1→32, yielding only ~300 MB/s despite a 12 GB/s link.
AI-written summary · about
chapters (9)
- 0:00ntprof introduction
- 1:19Disaggregated storage and NVMe-over-TCP
- 3:55Profiling NVMe-over-TCP challenges
- 6:47Limitations of profiling tools
- 8:14Modeling NVMe-over-TCP as network
- 10:04Trace points and data collection
- 12:15ntprof workflow and implementation
- 13:22Evaluating ntprof with use cases
- 15:50Summary and open-source release
related talks
17mHow to diagnose nanosecond network latencies in rich end-host stacks
29mProfiling Node Applications
18mMethodology and Practice in Observing Kernel Networking
15mNeutrino: Fine-grained GPU Kernel Profiling via Programmable Probing
34mTrace-Profiling: A New Way About How to Track Application Behavior
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
51mProduction Profiling: What, Why and How
37mKernel and userspace tracing with LTTng and friends