Kafka streams testing: A deep dive
summary
An attempt to dedupe across streams uses a reduce+filter topology that passes TopologyTestDriver unit tests but outputs nothing in production because the operator cache batches updates and only flushes on commit interval 30s, turning final values into the filtered “stop” token. Rewriting with a ValueTransformer and a persistent local store fixes the bug; integration tests (embedded Kafka or Testcontainers) catch the mismatch but run slower and can be flaky, so both test approaches are necessary.
AI-written summary · about
chapters (12)
- 0:00Introduction and welcome
- 3:05Overview of Kafka Streams testing
- 6:18Production issue and initial solution
- 13:09Topology test driver deep dive
- 19:19Why topology test driver differs from Kafka
- 24:57Kafka Streams execution loop
- 30:59Topology test driver execution model
- 36:00Topology test driver limitations
- 43:14Integration testing with embedded Kafka
- 53:59Optimizing integration test waits
- 1:03:12Choosing between test approaches
- 1:05:24Summary and Q&A
related talks
34mTesting Production Streaming Applications
30m🌴 Hooray Let's Dance Capoeira While Testing Our Kafka Streams 🕺 💃
44mTesting: Integration Tests - The good, bad and ugly
16mDiffStream: Differential Output Testing for Stream Processing Programs
46mOndrej Babec, Jiri Novotny: Performance Testing and Why Even the Imperfect One is Important
43mRemoving complexity from integration tests using Testcontainers!
42mDeciphering the way data is tested
15mAn Empty Kafka is a Sad Kafka