Finding and fixing a data-corruption bug...
summary
Rolling out PostgreSQL 16 causes 0.15% of instances to alert on read errors that trace to the free space map pointing past file end, amplified by a pg16 allocation change that batches page extensions; investigation uses WAL dumps, relation lookups and per-page checks, finds the FSM as the root, and mitigations include VACUUM FULL, removing the free space map, or an internal extension exposing pg_freespacemap to truncate safely; a lightweight check patch lands in 16.3 and is backpatched for customers.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 1:14Bug discovery in PostgreSQL
- 4:06Symptoms of data corruption
- 8:07PostgreSQL storage internals
- 10:18Free space map corruption
- 15:22Root cause analysis
- 19:03Fixing the free space map bug
- 21:31Key takeaways and lessons
- 23:21Q&A session
related talks
32mDebugging Data Corruption in PostgreSQL: A Systematic Approach
20mDebugging Data Corruption in PostgreSQL
52mData Corruption Bugs: Diagnosis and Lessons
45mEnhancing PostgreSQL Plasticity: New Frontiers in Memory Management
42mMy Journey in PostgreSQL bug fixing
01h 00mThe 10 Deadly PostgreSQL Sins
37mDatabase in Distress: Testing and Repairing Different Types of Database Corruption
01h 07mNavigating Challenging Customer Escalations in Postgres