Streaming Fast and Slow

on YouTube ↗38m

" summary

Practical guide to using metrics to shape streaming apps with Apache Flink and external reporters (Prometheus, Datadog). Emphasizes measuring watermarks, per-source throughput and per-reduce throughput, and using checkpoint size (steady growth with a daily drop) rather than just checkpoint count to spot lost state. Shows how Kafka lag patterns reveal upstream behavior, how relabeling alerts and routing to team Slack channels reduces noise, and how iterating dashboards and thresholds led to changed ownership of integration points.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and agenda
2. 1:48Data-driven basics for stream processing
3. 8:04Stream processing fundamentals
4. 13:59Where to start with metrics
5. 19:31Knowing your normal in metrics
6. 26:32Making metrics accessible
7. 28:18Metrics as shared language
8. 37:00Summary and Q&A

" related talks

• 38mFaster and Furiouser: Flink DriftTed Dunning · Flink Forward Berlin 2016
• 42mApache Flink Worst PracticesKonstantin Knauf · Flink Forward Europe 2019
• 15mStreaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful JobsShahid Chohan · Flink Forward Global Virtual 2020
• 34mA journey of scaling high-volume low-latency stateful processingArti Pande · Flink Forward Global Virtual 2020
• 26mHow Streaming Helps Your Staging Environment and Sandboxes Stay Up To DateMike Solomon · Flink Forward Virtual 2020
• 24mPushing Your Streaming Platform to the LimitElad Leev · Conf42 Chaos Engineering 2024
• 30mLive Coding: Building Stateful and Scalable Reactive Streams...Nolan Grace · Scale By the Bay 2021
• 51mMetrics are Not Enough: Monitoring Apache KafkaGwen Shapira · HighLoad++ 2017