Asynchronous I/O Stack: A Low-latency Kernel I/O Stack for Ultra-Low Latency SSDs

on YouTube ↗19m

" summary

An asynchronous I/O stack replaces synchronous kernel I/O steps with overlapped operations and a small set of engineering changes: per-core DMA-premapped page pool to avoid on-path DMA setup, deferring page-cache insertion until after submit, optional delayed IOMMU mapping, and a lightweight ldiot block layer that avoids merging and heavy allocations. A Linux prototype on NVMe/Optane achieves single-digit 4KB random read latency (7.6 µs), cuts overall I/O latency by ~15–33%, fsync latency by up to 26%, and boosts RocksDB IOPS 11–44% and Filebench/OLTP throughput 15–30%.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 0:31Storage Performance Trends
3. 1:29Synchronous I/O vs. Asynchronous I/O
4. 2:22Read Path Overview
5. 2:45Write Path Overview
6. 3:47Analysis of Vanilla Read Path
7. 5:07Asynchronous Page Allocation / DMA Mapping
8. 6:16Lazy Page Cache Insertion
9. 7:03Lazy DMA Unmapping
10. 7:50Linux Multi-queue Block I/O Layer
11. 9:58Light-weight Block / Layer
12. 10:10Read Path Comparison
13. 10:36Analysis of Vanilla Fsync Path (Ext4 Ordered Mode)
14. 12:08Fsync Path Comparison (Ext4 Ordered Mode)
15. 12:20Experimental Setup
16. 12:39FIO Performance (Random Read)
17. 13:35FIO Performance (Random Write Fsync, Ext4 Ordered)
18. 14:29Conclusion

" related talks

• 17mλ-IO: A Unified IO Stack for Computational StorageJiwu Shu, Zhe Yang, Youyou Lu, Xiaojian Liao, Junru Li, Youmin Chen, Siyu He · FAST '23: 21st USENIX Conference on File and Storage Technologies
• 22mI/O in a Flash: Evolution of ONTAP to Low-Latency SSDsRam Kesavan, Bharadwaj V. R., Matthew Curtis-Maury, Nikhil Mattankot, Vania Fang, Yash Trivedi, Kesari Mishra, Qin Li · 22nd USENIX Conference on File and Storage Technologies (FAST '24)
• 40mAchieving the Ultimate Performance with KVMBoyan Krosnov · SREcon20 Americas
• 19mP2CACHE: Exploring Tiered Memory for In-Kernel File Systems CachingHui Lu, Lingfeng Xiang, Zhen Lin, Jia Rao · USENIX ATC '23
• 21mStreamCache: Revisiting Page Cache for File Scanning on Fast Storage DevicesGuangyan Zhang, Zhiyue Li · USENIX ATC 2024
• 26mAnalysis of AMD HW-assisted vIOMMU Implementation and PerformanceWei Huang, Suravee Suthikulpanit · KVM Forum 2021
• 22mCombining Buffered I/O and Direct I/O in Distributed File SystemsAndré Brinkmann, Wei Xue, Yingjin Qian, Marc-André Vef, Patrick Farrell, Andreas Dilger, Xi Li, Shuichi Ihara, Yinjin Fu · 22nd USENIX Conference on File and Storage Technologies (FAST '24)
• 19mBest I/O is No I/OFrank Mashraqi · Velocity 2009