Handling frequent changes in state structure
summary
A utility that forces an operator to reset when its state layout changes is built and running in production for DataDome using Flink and Scala, with code at https://github.com/DataDome/flink-utils. It computes a UID from the operator id plus a hash of a state type signature that first uses Flink TypeInformation then falls back to developer-supplied signatures for non-case classes, with shapeless generation for case classes and support for Kryo-serialized fields. It prevents whole-job resets, requires a unit test to avoid accidental state loss, and works in practice.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:16Bot Detection
- 0:49Startup Mode
- 1:58Can't you just drop your state?
- 2:36What about state migration?
- 3:52Human Error
- 4:25Two ways to go
- 5:09Trade-offs
- 5:45What is TypeInformation? Flink's way of describing a type for serialization/deserialization
- 7:49Multiple States
- 8:29Generic Types
- 9:38Back to square one
- 10:12Requirements
- 10:35Type Signature on Types
- 10:56State Type Signature on Functions
- 12:45Building a TypeSignature
- 13:40Define Default Signatures
- 14:45Wrapping Up
- 16:17Questions!
related talks
16mFlink acceptance testing and state compatibility checking
34mTesting Production Streaming Applications
30mA Debuggers Guide to Apache Flink Streaming Applications
36mFlinkspector: Taming the squirrel
25mMastering UIs with Finite State Machines
13mUpFuzz: Detecting Data Format Incompatibility Bugs during Distributed Storage System...
01h 11mKafka streams testing: A deep dive
46mTesting in the world of Functional Programming