Secure and Efficient Data Sharing with Federated Cloud Storage
summary
A virtual data lake aggregates only metadata while original files remain in place, enabling on‑demand transfer and centralized access control; implementation combines Alluxio for federated data access and caching, LinkedIn DataHub for metadata/lineage, and Open Policy Agent with OPAL for distributed policy decisions. Use case covers cross‑company carbon‑emissions calculation and the architecture runs on Kubernetes; a simple download benchmark shows authorization adds minimal latency and the policy layer scales in parallel. Snowflake data sharing growth cited ~90–93% year over year.
AI-written summary · about
chapters (11)
- 0:00Federated cloud storage intro
- 2:09Cross-organization sharing challenges
- 4:15Virtual data lake benefits
- 6:48Implementation with open source
- 8:36Why AIO for federation
- 11:28Metadata and governance
- 15:46Data security and access control
- 18:36Fine-grained access with OPA
- 20:05Scaling OPAL enforcement
- 21:53Architecture performance
- 22:42Summary and future
related talks
15mDelta sharing: An open protocol for secure data sharing
27mConsistent Security, Governance And Flexibility For All Workloads
17mArchitecting a security data lake at enterprise scale
17mUnifying Your Security Data with Amazon Security Lake and OCSF
54mBuild your security data lake w/ Amazon Security Lake, featuring
13mSimplifying data security in a complex data lake environment
42mThreat Detection across all environments with SnowflakeData Security Lake
45mEnabling Data Sovereignty in AI Workflows through Data Mesh Architecture