Sources, Sinks, and Operators: A Performance Deep Dive

on YouTube ↗36m

" summary

Splunk describes scaling a Flink job to 20 GB/s (~1.7 PB/day) on a GCP Kubernetes fleet of 200+ n1-standard-32 VMs and thousands of task threads, and shows which knobs actually move throughput: Pulsar for replayable ingest, tuning max-parallelism to avoid key-group skew, using async profiler to find costly serializers, preferring parallel GC, and avoiding checkpoint alignment death-spirals. Practical wins include enable object reuse (+20% throughput), careful async IO sizing to avoid overwhelming sinks, and 3x batching via heartbeat-connected streams for sink throughput.

AI-written summary · about

" chapters (5)

1. 0:00Introduction
2. 3:57The basic challenges
3. 10:17Sources
4. 21:22Sinks
5. 25:10Putting it all together

" related talks

• 21mSplunk Data Stream ProcessorJamie Grier · Flink Forward Global Virtual 2020
• 34mA journey of scaling high-volume low-latency stateful processingArti Pande · Flink Forward Global Virtual 2020
• 47mBring Flink to Large Scale ProductionXiaowei Jiang · Flink Forward Berlin 2017
• 36mDynamically Generated Flink Jobs at ScaleRegina Chan · Flink Forward Europe 2019
• 31mFlink @WORKDAY: scaling of ingestion data pipelines to S3Leire Fernandez de Retana, Enrico Agnoli · Flink Forward Global Virtual 2020
• 43mLarge Scale User Behavior Analytics by FlinkHao Wu · Flink Forward Berlin 2017
• 43mTo Petascale and Beyond: Apache Flink in the CloudsGreg Hogan · Flink Forward Berlin 2016
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020