IO and cgroups, the Current and Future Work
summary
Facebook deploys a kernel-layer controller that enforces per-cgroup latency targets by delaying requests before they become hardware outstanding, uses blk_iolatency plus a user-space replacer for the OOM killer called OMD, and adds a proportional mode called IO.weight. It prevents IO-driven memory thrash (unprotected boxes take ~50 minutes to recover, protected ones recover across repeated stress runs in minutes), sacrifices about ~10% bandwidth worst case, delays throttling to avoid lock inversions, and avoids ext4 ordered-mode priority inversions by moving to btrfs.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:31Why do we need a io controller?
- 1:09Background
- 2:19The io stack (multi-queue)
- 4:41Challenges and solutions
- 8:30Memory leak on spinning rust
- 10:18Memory leak on SSD
- 11:0010 storm with tar on spinning rust
- 11:50Fbtax2 slices
- 13:04Fbtax2 components
- 13:59io.weight
related talks
25mRecent developments in I/O control and testing
42mcgroup2 Resource Control strategies w/ resctl-demo
27mThe Two New I/O Controllers and BFQ
41mA Solution to High Latencies Caused by I/O
36mNew Cgroup Subsystem for Buffer Write io and Network RX Control in Kernel
38mUnderstanding and Working with the Cgroups Interface
59mThe Wild West of post-POSIX IO Interfaces
36mNew Cgroup Subsystem for Buffer Write io and Network RX Control in Kernel