Five data lake considerations
summary
Five practical rules: give every dataset a minimal schema so analysts can run Athena/Presto/SQL queries immediately (data is read many more times than written), build reusable plumbing like event stream compaction and upsert consolidation plus pipeline monitoring so onboarding becomes configuration not custom dev, let analysts create materialized views with Redshift Spectrum rather than waiting for engineers, and follow partitioning, columnar compression, scheduled compaction and S3 server-side encryption plus IAM, bucket policies and CloudTrail for performance and security.
AI-written summary · about
chapters (7)
- 0:00Intro
- 0:40Avoid the swamp: Make your lake truly analytics optimized
- 3:11Enable exploration by giving all data a minimal schema
- 6:40Strive to reduce or eliminate engineering in data curation
- 10:10Make materialized view creation self-serve for analysts
- 11:06Unify data warehouse and data lake
- 13:52Use the right primatives to secure your data
related talks
43mManaging a Data Lake on AWS
34mBest Practices for Implementing a Data Lake in Amazon S3
43mSoftware Quality As It Relates To Data
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
28mYour data lakes need architecture too!
34mBest Practices for Implementing a Data Lake in Amazon S3
55mDesigning performant and scalable data lakes
56mUnderstanding and achieving a modern data architecture