High performance Privacy By Design using Matryoshka & Spark
summary
Built a privacy framework that applies semantic-tagged strategies to nested JSON-like records and ships three concrete engines: a straight Matryoshka zip-of-schema-and-data engine, a precomputed lambda-on-heap transformer, and a Spark-integrated Catalyst codegen producing unsafe Tungsten bytecode. Benchmarks on a 10-node cluster: ~70 → ~45 → ~21 minutes respectively. The system compiles Java on the driver, ships bytecode to executors, avoids GC churn, and is practical — implemented end-to-end and used in production.
AI-written summary · about
chapters (6)
- 0:00Start
- 3:47Privacy Framework
- 8:11Our Data Structure
- 25:59Matryoshka Privacy Engine
- 35:43Codegen Privacy Engine
- 42:40Performance trial !
related talks
43mSpark Over a Mesos Secured Productive Environment
54mSecuring Software by Construction
39mData Privacy Engineering to power Responsible Enterprise AI
20mGeneralized Policy-Based Noninterference for Efficient Confidentiality-Preservation
31mLeverage AI to Protect Your Data Assets... from AI
37mScaling and Securing Spark on Kubernetes at Bloomberg
50mUsing language-theoretics and runtime visibility to align AppSec with DevOps
29müberSpark: Enforcing Verifiable Object Abstractions for Automated...