Evolving Yelp's Streaming at Scale

on YouTube ↗37m

" summary

Yelp runs streaming on Flink with a custom control plane, schema registry and CLIs, and currently operates 50 Flink services across 250 clusters running roughly 2,600 jobs. The platform moves from many homegrown components (custom CDC, DataPipelineMessage, Yelp schematized, session clusters) toward a single SQL client, Flink upgrades, and open formats/registries to enable Streamhouse-style separation of compute and object storage while solving autoscaling, state compatibility, and S3 checkpoint retention.

AI-written summary · about

" chapters (10)

1. 0:00Introduction to streaming at Yelp
2. 1:23Streaming ecosystem overview
3. 7:27Early data processing with MapReduce
4. 9:12Building data pipeline tools
5. 12:02Introducing Stormboard for streaming
6. 14:10Transitioning to Apache Flink
7. 18:29Expanding platform options
8. 23:26Platform challenges and strategies
9. 30:34Future plans and open source alignment
10. 33:46Q&A session

" related talks

• 21mFlink's application at DidiXue Kang · Flink Forward Virtual 2020
• 30mKubernetes + Operator + PaaSTA = Flink @ YelpAntonio Verardi · Flink Forward Europe 2019
• 47mBring Flink to Large Scale ProductionXiaowei Jiang · Flink Forward Berlin 2017
• 43mLeverage the power and simplicity of SQL on Apache FlinkFabian Hueske · Berlin Buzzwords 2018
• 20mBuilding metric platform using Flink for massive scale at NetflixAbhay Amin · Flink Forward Virtual 2020
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · DataEngBytes 2021
• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016