Fast data processing in a data lake with SQL

on YouTube ↗51m

" summary

Describes using distributed SQL engines—Trino, Presto and Dremio—to run SQL queries directly over files in a data lake, reducing ETL and enabling more flexible, efficient analytics. Covers the advantages of querying data-in-place from the lake and how these engines execute SQL against file formats stored there.

AI-written summary · about

" chapters (18)

1. 0:00Introduction
2. 1:29Analytical systems evolution
3. 4:40Compute-storage separation
4. 7:51Query optimizers in engines
5. 9:16Column pruning and columnar data
6. 9:37Filter pushdown and data spin
7. 12:51Dynamic filters in joins
8. 16:24Massively parallel parallelism
9. 18:32Optimal file partitioning
10. 21:03Virtual partitioning
11. 24:56Metadata and data caching
12. 29:13Format decoding problems
13. 32:22Decoded data caching
14. 36:13Compute-storage combination
15. 38:21Intermediate results caching
16. 40:38Materialized views
17. 43:49Summary and success stories
18. 44:11Q&A

" related talks

• 22mIntroduction to Presto: The SQL Engine for Data Platform TeamsRohan Pednekar, Philip Bell · Emerging OS Forum @ Open Source Summit North America 2022
• 02h 05mBig Data and the Data Lake from the Analyst PerspectiveScott Shaw · PASS Business Analytics Conference 2015
• 22mPresto for Your Fast, Open SQL LakehouseSteven Mih · OSA Con 2022
• 22mBig data toolsPaul Singman · Global SRE Virtual Conference 2022
• 27mHarnessing the Power of Dual Query Engines.Lei Gao, Jeff Xu · Scale By the Bay 2023
• 43mData Lake beyond the BuzzwordMatheus Espanhol · QCon São Paulo 2018
• 06mDriving enterprise open source adoption, from data lake to AIRon Bodkin · Strata + Hadoop World 2017
• 51mHow to Build a Modern Data Lakehouse?Piotr Kalinowski · 4Developers 2022 Online