GDPR Data Cleaner: Mutating Immutable Data
summary
GoPro builds a configuration-driven GDPR data cleaner that lets teams delete user records from big data without replacing the platform: it uses Spark SQL insert overwrite to implement delete-by-filter on Parquet partitions, supports Parquet and Hive tables plus HBase, performs manual/automatic compactions, keeps backups and rollback, and handles email-to-ID translations. Implementation focuses on delete-only mutations, scales with Spark, takes about three to six months to reach production, and is planned to be open sourced at github.com/gopro.
AI-written summary · about
chapters (10)
- 0:00Introduction to GDPR data deletion
- 1:07GoPro data platform overview
- 3:34GDPR and CCPA basics
- 5:18Challenges with immutable data
- 8:05Mutable big data storage options
- 11:58GoPro’s data cleaner tool design
- 16:31Spark SQL delete implementation
- 19:43Data cleaner architecture
- 22:52Summary and open source plans
- 24:18Q&A session
