Data Lake beyond the Buzzword
summary
An experienced data engineering lead describes how to make Data Lakes deliver business value rather than become useless. The talk reviews trade-offs, errors and successes from Movile’s PoCs and production rollout, comparing tools for storage and distributed processing (Amazon S3, Presto, Spark, Redshift Spectrum, AWS Athena, BigQuery) and for data/metadata management (AWS Glue, Apache Atlas, Dremio, Kylo). It focuses on practical architecture and implementation challenges when assembling open-source and cloud services for Big Data.
AI-written summary · about
chapters (18)
- 0:00Data Lake introduction
- 1:18Why implement a Data Lake
- 4:13Cloud solutions and Redshift
- 5:16Entering the Data Lake world
- 7:27Data Lake architecture on S3
- 10:00Structure and technologies used
- 13:11Solving specific problems
- 16:30Testing Presto and BigQuery
- 18:36Performance and cost challenges
- 21:03BigQuery feedback and improvements
- 22:50Adopting Athena for analysis
- 24:59Data Lake layers
- 27:09Delivering agile solutions
- 29:17User profiles and tools
- 31:45Automating metadata
- 34:40Data governance and security
- 37:31Lessons learned and common errors
- 40:37Process review and insights
related talks
17mOpen Data Lakes: Latest stop in the big data journey
40mThe Journey to Implementing a Data Lake
42mSolve Big Data Challenges with Data Lake
51mHow to Build a Modern Data Lakehouse?
43mManaging a Data Lake on AWS
41mOn Data: Sky above, sand below, peace within
45mThe Data Lakehouse: A tech talk for everyone who loves data
55mDesigning performant and scalable data lakes