Flink's application at Didi

on YouTube ↗21m

" summary

Built a Flink-based real-time platform and a SQL layer that lets non-experts write streaming jobs; StreamSQL based on Flink Table API runs alongside custom schedulers and resource-tuning that adapt configs from observed CPU and memory use. Platform serves hundreds of teams with 1000+ Hadoop nodes and ~3000 Flink jobs (about 2000+ SQL jobs) processing 2 trillion records per day, with Kafka sources and sinks to HDFS, HBase, Elasticsearch and ClickHouse; challenges are large state IO, HA recovery and multi-language support.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 0:27Guideline
3. 0:59History of stream computing at DIDI
4. 3:38Business Scenarios
5. 6:13Multi-Cluster Architecture
6. 8:30Advantages of StreamSQL
7. 10:02Refined DOL
8. 12:14UDX Extension
9. 13:26Stream Join
10. 15:04StreamSQL DE
11. 15:46Job Management
12. 16:44Job Operations
13. 17:55Stream Meta Construction
14. 19:31Challenges
15. 20:37Future Plan

" related talks

• 43mLeverage the power and simplicity of SQL on Apache FlinkFabian Hueske · Berlin Buzzwords 2018
• 47mBring Flink to Large Scale ProductionXiaowei Jiang · Flink Forward Berlin 2017
• 39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Data Con LA 2021
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · DataEngBytes 2021
• 41mBeyond the Hype: When Apache Flink Solves Real ProblemsNaci Simsek · Berlin Buzzwords 2026
• 32mThe easy way to build and run Flink SQL jobs using Cloudera SQL Stream BuilderKenny Gorman · Flink Forward Global 2021
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020