Keeping data private in real-time pipelines
summary
Practical pipeline patterns for keeping streaming data useful while preventing reidentification using Apache Kafka and Apache Flink with Iceberg storage: deterministic versus random masking, replacing PII with a token vault, why hashes fail (Netflix Prize and New York taxi MD5 reversals) and how aggregated noise or k-anonymity and bucketing prevent leaks, plus synthetic data and “shadow traffic” for testing; shows live pipeline demos that anonymize before writing to storage and concrete tradeoffs between utility and privacy.
AI-written summary · about
chapters (9)
- 0:00Data privacy in pipelines
- 1:51Removing identifiers isn’t enough
- 5:24Privacy risks in streaming
- 8:11Data masking and tokenization
- 15:34Re-identification dangers
- 22:20Stronger privacy techniques
- 25:27Noise and synthetic data
- 28:59Balancing usefulness and sensitivity
- 31:05Privacy protection strategies
related talks
40mKeeping data private in real-time pipelines
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
50mPrivacy by design
38mInteractive Session: Optimising data streaming pipelines on Flink and Kafka
16mLife is But a (Data) Stream: Building Quality Data Pipelines
26mArchitecting real-time transactional data lake
43mSharing responsively: Streaming analytics from sensitive data
11mBig Data Analytics: Immutability & Idempotence