In-depth technical story: Fixing I/O performance for Windows guests in OpenStack Ceph clouds
summary
A private OpenStack cloud with Ceph shows Windows VMs stalling because many writes are unaligned <4KB partial writes; 1.4% of operations consume ~40% of CPU time and stall two OSD threads, producing multi‑second latencies. Investigation uses perf flame graphs and custom Ruby parser plus Ceph perf dumps and dump-ops-in-flight. Fixes touch Ceph (drop posix readahead/advise), Nova/Cinder metadata, and Windows VirtIO drivers; patches reduce partial writes from 23% to 1.8%, cut disk reads ~40→2/s and eliminate ~75% of the slow ops.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:56User Complaints
- 5:37ceph daemon osd. 1 perf dump (latency)
- 6:33ceph daemon osd. 1 perf dump (throttle)
- 7:55ceph daemon osd. 1 dump_ops_in_flight
- 11:07ceph tell osd.* config set debug_filestore 10
- 17:19Flamegraph
- 21:25How FileStore works
- 25:20Possible Fix #1
- 25:53Reproduction
- 27:33madvise(FADVISE_DONTNEED)
- 28:53Possible Fix #2
- 29:53512b/4k Disk Alignment ("4K Drives")
- 33:17Additional Fix #3
- 34:09Additional Fix #4
- 35:45Additional Fix #5
- 37:36Key Takeaways
related talks
26mLessons Learned: Optimizing KVM Performance for EHR Systems
35mWhy OCP networking performances on VMs are suboptimal and how to improve them by 5x
50mOptimizing for NVMe Drives: The 10 Microsecond Challenge
20mChasing Bad Checksums: A Journey Through Ceph, TCMalloc, and the Linux Kernel
28mCeph on Windows
39mUsing EBPF to Develop a Powerful Tool for Diagnosing Ceph Performance Issues
21mKeynote: Ceph Developers AMA: Meet the Minds Behind Open Source Ceph
47mExploring Linux Memory Usage and IO Performance in Cloud Native Applications