Recovering from a damaged cluster
summary
A real case recovers a 2 tera-byte database corrupted by bad RAM by booting Postgres with zero_damage_pages to ignore damaged blocks, using single-user mode to alter system catalogs, and dropping and recreating or reindexing corrupted sequences and indexes. The cluster is dumped off that running instance, accepting that some rows with damaged pages are lost, and a careful vacuum/reindex pass cleans associations. The method succeeds but requires regression testing and manual vetting of returned records.
AI-written summary · about
chapters (8)
- 0:00Start
- 9:25System Catalogs
- 11:24How Does the Manifestation Breaking Show Up
- 18:23The System Index
- 28:19Plan B
- 28:38Guilt by Association
- 31:01Executing a Dump
- 33:36Toasted Tables
related talks
20mDebugging Data Corruption in PostgreSQL
51mDatabase in Distress: Testing and Repairing Different Types of Database Corruption
37mDatabase in Distress: Testing and Repairing Different Types of Database Corruption
42mHow to corrupt your database (and how to deal with data corruption)
32mDebugging Data Corruption in PostgreSQL: A Systematic Approach
42mPostgres: stories that will scare you
32mPostgreSQL The Hard Way
38mWhen it All Goes Wrong: Database Incident Response Dos and Don'ts