Privacy by design
summary
Explains concrete engineering patterns to meet GDPR: use a workflow orchestrator so batch Spark/MapReduce pipelines can reprocess raw events and honor deletion requests (two large companies already do this), avoid personal IDs as keys, prefer splitting personal fields into a small table and the rest into the data lake (ejected record) or encrypt fields and discard keys (lost key); warn about Cassandra tombstone deletion issues and aim for short retention windows (roughly one month) and careful pseudonymisation.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:15Presentation
- 1:35Privacy by design
- 3:06Tools
- 3:55Challenges
- 5:12Digital Stone Age
- 5:58Modern Data Processing
- 9:53Workflow Orchestrator
- 12:06Data Processing Architecture
- 13:37Technical Solutions
- 14:10Classification
- 18:11Make it easy
- 20:57Privacy in a data lake
- 21:44Privacy in a relational database
- 22:40Consent
- 26:35Right to be Forgotten
- 27:12Data Discovery
- 34:19Objective Record Pattern
- 38:13Workflow
- 41:27Tombstones
- 42:07Deleting data
- 46:16Removing data
- 46:29Streaming
related talks
19mData etiquette in a regulated world
28mData Engineers: Privacy Is Your Problem
39mdev.privacy: GDPR in a nutshell
39mData Privacy Engineering to power Responsible Enterprise AI
18mDesigning for data privacy
40mGDPR in practise - Developing models with transparency and privacy in mind
40mData Privacy and Security by Design
01h 59mDeep Dive Workshop – Data Security and Data Protection