Interactive Session: Optimising data streaming pipelines on Flink and Kafka
summary
Practical tips for running Flink with Kafka: watch for skewed partitions—one hot partition can break both Kafka and Flink; fix producer keying or rebalance partitions. Tune broker settings like socket.receive.buffer.bytes, message/batch sizes and max request queues to soften black‑swan spikes. Prefer building a packaged job (Table API/JAR) for production rather than the SQL CLI; SQL client needs better diagnostics and SQL unit testing is hard, so use in‑memory or Docker integration tests. Consider differential privacy for GDPR on grouped outputs.
AI-written summary · about
chapters (9)
- 0:00Introduction and panel setup
- 2:24Panelists' Flink backgrounds
- 5:57Kafka and Flink integration
- 8:29Skewed data challenges
- 14:07Missing features in Flink
- 20:09SQL in production
- 22:59SQL vs Table API
- 29:04Alternatives to Flink
- 33:43Advice for new users
related talks
27mNavigating Streaming Infrastructure
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
29mWhen your stream doesn't stream very well
19mIntegrating Flink and Kafka in the standard genomics pipeline
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
41mBeyond the Hype: When Apache Flink Solves Real Problems
46mFlink SQL in 2020: Time to show off!
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...