Presto - Today and Beyond - The Open Source SQL Engine for Querying all Data Lakes
summary
Presto is an open source Apache 2.0 SQL engine that queries data lakes in place, using a coordinator-worker architecture that runs at Facebook scale (up to ~1,000 workers) and reads ORC or Parquet on S3/HDFS. Connectors (Hive connector for file stores, many DB and streaming connectors) allow federated joins. Long ETL runs can use Presto on Spark to gain Spark’s shuffle and recoverability, while RaptorX cache and materialized views deliver up to ~10x query speedups. Spilling, external spill plugins, coordinator segregation and external UDF servers improve reliability and extensibility.
AI-written summary · about
chapters (38)
- 0:00Introduction
- 0:56What is Presto
- 3:50Architecture
- 5:16Common Questions
- 8:02Presto Architecture
- 9:35Presto CLI
- 11:40Presto connectors
- 12:12Presto connector model
- 14:08Hive connector types
- 15:27Speed
- 16:18Query
- 17:04Aggregate
- 17:38Use Cases
- 18:50Stacks
- 19:38Data Science
- 20:04Federated
- 20:41Transformations
- 21:44Data Lakes
- 23:32Other Query Engines
- 24:58HANA
- 26:07Technical Improvements
- 26:50Reporting Dashboarding
- 27:22Interactive Analytics
- 27:41High QPS
- 28:21Large workloads
- 29:22Consolidate on Presto
- 30:13Scaling Presto
- 32:52Workload
- 35:01Online Analytics
- 35:31Cache Architecture
- 37:21Presto Plus
- 37:43Improving Reliability
- 38:37Improving Task Scheduling
- 39:29ClientSide Retries
- 39:52Native Accelerator
- 41:20AriaScan
- 41:47Extensibility
- 43:50Get involved
related talks
22mIntroduction to Presto: The SQL Engine for Data Platform Teams
22mPresto for Your Fast, Open SQL Lakehouse
28mOptimizing Latency-Sensitive Queries for Presto at Facebook
32mData @Scale Seattle- Eric Hwang introduced by Ganapathy Krishnamoorthy
31mPanel: Open Data Lakes
40mBreakout Session
27mHarnessing the Power of Dual Query Engines.
10mPresto-Native Noisy Aggregations for Privacy-Preserving Workflows