ETL - Extract Trino Load - A Case for Trino as a Batch Processing Engine
Talkat Open AI + Data Forum @ Open Source Summit North America 2022 · on Wednesday, 2022-06-22· at 11:00· Room 408/409 (Level 4), JW Marriott Austin, 110 E 2nd Street Austin, TX 78701·
summary
Argues Trino can be a fault-tolerant batch ETL engine by adding an exchange buffer so tasks run and recover independently instead of streaming everything at once. Implements a file-system buffer proof-of-concept (S3/GCP/Azure, up to 50 partitions) and is building a layered memory/SSD buffer. Demonstrations include a 5 TB memory query on single node by iterative task scheduling, TPCH 10 TB with 20 concurrent queries on five nodes without OOM, and community reports of ~60% cost savings with spot autoscaling.
AI-written summary · about
chapters (7)
- 0:00Start
- 7:07Diagnosing a slow running query
- 11:49ETL: Challenges
- 17:44ETL: Community Adoption Challenges
- 26:24FTE: Resource Allocation
- 31:16Distributed Buffer: Basic Implementation
- 41:33FTE: Adaptive Optimizations
related talks
34mCloud Agnostic Data Platform with DBT, Trino, Iceberg and MinIO
27mHarnessing the Power of Dual Query Engines.
51mFast data processing in a data lake with SQL
16mETL vs ELT: Why the T has shifted
38mReigniting the ETL vs. ELT Debate - Which Better Supports the Business?
15mNext Gen ETL Generators
30mThe Incremental ETL Architecture
05mDBTB INT Shinji Kim