Building Presto SQL experience on Flink

on YouTube ↗34m

" summary

Adds Presto SQL as a dialect on Flink inside AthenaX by translating Presto ASTs into Calcite/Flink plans and by optionally using Presto codegen; implements explicit query rewrites to emulate tumble/hop/session windows (creates a window_assign UDF and flattens window arrays via cross joins), maps and migrates UDFs with tooling such as Apache Coral, and validates queries against Flink 1.12. Works end-to-end for usability (debugging and interactive runs) at Uber scale—~2,300 streaming pipelines, ~3 trillion events/day, Presto ~12k users and ~400k queries/day—performance tuning remains future work.

AI-written summary · about

" chapters (4)

1. 0:00Overview
2. 7:19Default SQL
3. 12:58Implementation
4. 32:37Future work

" related talks

• 22mIntroduction to Presto: The SQL Engine for Data Platform TeamsRohan Pednekar, Philip Bell · Emerging OS Forum @ Open Source Summit North America 2022
• 51mPresto Open Source SQL Engine for Massive Scale Data LakehousesJames Cho · Open AI + Data Forum @ Open Source Summit North America 2023
• 22mPresto for Your Fast, Open SQL LakehouseSteven Mih · OSA Con 2022
• 43mLeverage the power and simplicity of SQL on Apache FlinkFabian Hueske · Berlin Buzzwords 2018
• 50mPresto: SQL-on-Anything using KubernetesKarol Sobczak, Michael McCune, Matt Fuller · DevConf.CZ 2020
• 47mTaking a look under the hood of Apache Flink’s relational APIsFabian Hueske · Flink Forward Berlin 2016
• 39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Data Con LA 2021
• 37mSQL Ask-Me-AnythingBowen Li, Timo Walther, Kurt Young, Dawid Wysakowicz · Flink Forward Europe 2019