Of Data Dropboxes and Data Gloveboxes
summary
Claim: isolate sensitive data while still letting analysts compute on it. Uses Hadoop primitives—HDFS REST with checksums for Data Dropbox put only semantics, Kerberos auth and a servlet filter implementing a Dropbox filter that returns 403 on reads, Apache YARN for workload isolation, Oozie/Git for controlled exfiltration, PAM namespaces and Chef to enforce per-user temp dirs, iptables/net_cls (or Calico on Kubernetes) to block unwanted egress. Bloomberg reports it runs at scale (100 billion ticks and 1.5M stories a day) and the code is available open source.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:25Bloomberg By the Numbers
- 3:23Handling Material
- 3:54Properties of Radioactive Materials
- 5:18Properties of Data
- 6:13Nuclear Material (non) Proliferation
- 6:37Data Material Proliferation
- 9:20Clean-Up Is Messy
- 9:46Hadoop Cluster
- 12:18Lock Everything Down
- 15:15DropboxFilter for HDFS WebHDFS API
- 16:05DropboxFilter for HDFS (Architecture)
- 17:28DropboxFilter for HDFS (Examples)
- 18:17Plutonium Enclave
- 19:51Data Enclave
- 21:26Isolation Glovebox
- 22:07Glovebox Production Line
- 25:09Data Glovebox (Leaded Pane of Glass)
- 26:02Data Glovebox (Glove Ports & Robotics)
- 27:17Data Glovebox (Pass-Through)
- 29:02Firewalls (Workload and User Isolation)
- 31:01Take out the Trash
- 31:40Private Temporary Directories
- 33:32Keeping the Pipes Flowing
- 35:36Firewalls Are Important
related talks
50mBabar-ians at the Gate: Data Protection at Massive Scale
31mRings of Security for Big Data in Cloud
26mAnalytics & Data Science with Privacy at its Core
24mSecure and Efficient Data Sharing with Federated Cloud Storage
10mAdding Security to Hadoop
33mSo you think you can airgap? (No.)
02h 05mBig Data and the Data Lake from the Analyst Perspective
09mData: Open for Good and Secure by Default