Flink acceptance testing and state compatibility checking

on YouTube ↗16m

" summary

Yelp builds a docker-compose based Flink compose sandbox that spins up a standalone Flink cluster, Kafka, Schema Registry and test fixtures to run end-to-end acceptance tests; tests create topics, push tailored input, submit jobs, tail output and assert results. CI/CD adds a state-compatibility step that takes a job, cancels to a savepoint, and resumes a new version to catch breaking state restores (Flink 1.6 behavior); one March 17 change blocked nightly deploys until a manual fix. Future work targets test templates, automated message generation and Flink State API migration.

AI-written summary · about

" chapters (3)

1. 0:00Intro
2. 5:20Lessons
3. 13:40State restoration

" related talks

• 50mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Jfokus 2025
• 25mFlink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8sMichas Szacillo, Wang Li · KubeCon + CloudNativeCon Europe 2025
• 41mRunning Apache Flink everywhereStephan Ewen · Flink Forward Berlin 2016
• 37mThe Future of Apache FlinkAljoscha Krettek · Flink Forward Berlin 2016
• 31mFlink @WORKDAY: scaling of ingestion data pipelines to S3Leire Fernandez de Retana, Enrico Agnoli · Flink Forward Global Virtual 2020
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 36mFlinkspector: Taming the squirrelAlexander Kolb · Flink Forward Berlin 2016
• 41mIntroducing Flink on MesosEron Wright · Flink Forward Berlin 2016