Optimizing for NVMe Drives: The 10 Microsecond Challenge
summary
Claim: NVMe drives hit 10 microsecond latency so software overhead dominates; bare-metal PCI device assignment with polling (Linux NVMe iole) reaches about 120k IOPS for 4KB Q=1 reads. KVM/QEMU without tuning starts near 21k IOPS; enabling IO threads, vfio PCI passthrough, multi-queue, packed virtqueues and a user-space NVMe driver with poll-mode completion and an AIO fast path raises VM IOPS to ~94–100k and narrows the gap. A qemu storage daemon is built to share user-space drivers across guests.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 0:33NVMe Drives
- 1:27Latency Numbers
- 2:46IOPS and Latency
- 6:49Rethink the architecture
- 7:58IO request lifecycle
- 10:00Latency
- 12:33Completion mechanisms
- 15:40PCIe device assignment
- 19:23Configuration
- 21:07libvert
- 22:13Example
- 24:55CPU Idle Halt Poll
- 27:20Results
- 28:51iopoll
- 30:00Bare metal vs NVMe
- 30:53Virtio Block
- 33:47IOThreads
- 35:39IOThreads Configuration
- 35:54Userspace NVMe Driver
- 37:35Polling Queues
- 39:19AIO Fast Path
- 41:00Linux IO Poll
- 45:41QMU Storage daemon
- 47:55Future direction
- 48:13Summary
- 48:47Conclusion
related talks
21mNVMe Emulation Performance Optimization
24mOptimizing Storage Performance for 4–5 Million IOPs
17mHow advancements in storage are triggering innovations in open-source and vice-versa
42mUnderstanding the Linux Block Layer-From Single Queue to Multi-Queue Architecture
19mI/O Passthru: Upstreaming a flexible and efficient I/O Path in Linux
33mHow Fast Can We Go? Booting a Linux VM to Userspace in 100ms and Beyond
39mSecured I/O in the Cloud with NVMe-Over-TCP
26mLessons Learned: Optimizing KVM Performance for EHR Systems