Testing Production Streaming Applications

on YouTube ↗34m

" summary

Shows how to test Flink from unit to full integration using Flink’s operator test harnesses for stateful UDFs, simple collection sources/sinks for batch-style pipeline tests, a homegrown manual pipeline tests utility (in the cloudera/flink-tutorials repo) to feed single events and watermarks and trigger timers, and embedded Kafka for connector tests. The manual tool works and exposes timing bugs but can miss production race conditions; integration tests against real connectors are recommended before deploy.

AI-written summary · about

" chapters (8)

1. 0:00Intro
2. 0:36Application Components
3. 1:58User Functions
4. 4:20Stateful Operators
5. 8:05Pipeline Testing
6. 15:46Manual Pipeline Tests
7. 21:29Integration Tests
8. 29:36Summary

" related talks

• 30mA Debuggers Guide to Apache Flink Streaming ApplicationsAlexander Fedorov · Flink Forward Global 2021
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 34mIntegrating Flink into Your Ecosystem - How to Build a Flink Connector from ScratchArvid Heise · Flink Forward Global 2021
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Data Con LA 2021
• 42mStream Processing beyond Streaming DataStephan Ewen · ITNEXT Summit 2019
• 36mFlinkspector: Taming the squirrelAlexander Kolb · Flink Forward Berlin 2016
• 27mIntroduction to Flink in 30 minutesDavid Anderson, Johannes Moser · Flink Forward Global 2021