Building Stateful Streaming Pipelines

on YouTube ↗19m

" summary

Built a low-latency streaming data platform at godaddy using apache beam and flink on EKS that cuts data availability from 24–48 hours to near real time; ingest uses Amazon Kinesis (7‑day retention) with firehose to S3 cold storage, per‑pipeline Flink clusters on Kubernetes, RocksDB for state, and savepoint-based node rotation plus blue‑green deploys. Example pipeline: fixed 1s windows with 1s allowed lateness to emit searches per second metrics; same Beam code also runs on Spark for batch, demonstrating portable pipelines.

AI-written summary · about

" chapters (6)

1. 0:00Introduction and motivation
2. 3:36Lambda vs Kappa architecture
3. 9:10Platform architecture overview
4. 13:44Pipeline authoring and abstractions
5. 15:13Deployment lifecycle and scaling
6. 17:19Key learnings and recommendations

" related talks

• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 32mStream Processing at Lyft with Flink and BeamSherin Thomas, Micah Wylde · Scale By The Bay 2018
• 46mBuilding an Apache Beam streaming runtime in GCPSergei Sokolenko, Reuven Lax · Flink Forward Europe 2019
• 28mSweet Streams Are Made Of TheseCaito Scherr · DataEngBytes 2020
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016
• 16mYour Data Pipeline with Apache BeamSouhaib Guitouni · Devoxx France 2021