Developing IO Prefetching for Postgres
summary
A feedback-driven prefetcher is developed and evaluated with a custom prefetch simulator that models io_uring submission/completion rings, a max IOPS + latency device, and variable consumption rates. The algorithm samples the rate of change of completed not consumed buffers and adjusts prefetch size/frequency; simulated results hit ~40 ms total scan latency with completed backlog ≈20 (vs ~100 ms for sync, or large backlogs when fetching everything). Work-in-progress: tuning ramps, fast vs slow storage multipliers, and upstream implementation for direct and asynchronous IO.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:54What is prefetching?
- 5:11Prefetching development issues & challenges
- 6:24Advantages of simulation
- 14:05Evaluating prefetching simulation output
- 23:21Algorithm work-in-progress
- 24:05Next steps
- 25:24Q&A with Melanie & Marco
related talks
01h 06mIO in PostgreSQL: Past, Present, Future
48mThe path to using AIO in postgres
49mStreaming I/O and vectored I/O
49mAIO in PG 18 and beyond
26mAdditional IO Observability in Postgres with pg_stat_io
48mStreaming I/O and vectored I/O
53mVisualizing Postgres I O Performance for Development
41mLinux IO internals for PostgreSQL administrators in 2023