Large Scale Stream Processing with Blink SQL at Alibaba

on YouTube ↗27m

" summary

Alibaba runs Blink SQL (Flink-based) as a production stream SQL engine after ~2 years of work, powering search, recommendation and ads and running most real‑time jobs on 2017 Singles Day with per‑job state in the multi‑terabyte range. Built features include dynamic tables and stream‑table duality, temporal (lookup) joins and a DDL/DML surface, latency‑SLA and state‑TTL controls, micro‑batching and top‑N pushdown optimizations, plus a UI ML platform that translates workflows to TableAPI/SQL.

AI-written summary · about

" chapters (16)

1. 0:00Introduction
2. 0:58Background of Alibaba
3. 2:37Alibaba Ecosystem
4. 4:10Why Scala
5. 5:10Dynamic Table
6. 7:04Stream Cycle
7. 8:56DDL
8. 9:54DML
9. 14:00Query Optimization
10. 16:10Summary
11. 17:18QA
12. 17:53Machine Learning
13. 18:30Stream Composition
14. 21:17Application aggregation
15. 22:47Optimization
16. 23:28Questions

" related talks

• 43mLeverage the power and simplicity of SQL on Apache FlinkFabian Hueske · Berlin Buzzwords 2018
• 23mWhat is Behind Alibaba's Double 11 Shopping FestivalYuan Mei · Systems @Scale Summer 2021
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 21mFlink's application at DidiXue Kang · Flink Forward Virtual 2020
• 39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Data Con LA 2021
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · DataEngBytes 2021
• 37mSQL Ask-Me-AnythingBowen Li, Timo Walther, Kurt Young, Dawid Wysakowicz · Flink Forward Europe 2019
• 49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing ApplicationsStephan Ewen · Scale By the Bay 2017