Panel: Large Scale Streaming in Production
summary
Companies report running Flink at production scale: millions of messages per second ingested, Netflix processes 1.4 petabyte/day, and teams at Alibaba and Uber use streaming for real-time recommendations and online ML. The panel discusses concrete pain points: operator-local state backed by RocksDB, async I/O for external lookups, incremental checkpointing and fast failover for large state, tradeoffs between queryable external stores versus in-engine state, and offering both SQL-style and programmatic APIs for different user groups.
AI-written summary · about
chapters (8)
- 0:00Introduction to panel
- 1:05Panelist introductions
- 4:18Drivers of stream processing adoption
- 16:00Stateful stream processing challenges
- 31:38Programming models for users
- 38:25Performance vs correctness in streaming
- 42:17External state access in pipelines
- 49:29Future of stream processors
related talks
37mBuilding and operating large-scale streaming applications at King
58mBuilding Self Service Platforms for Apache Flink
38mBuilding Unified Streaming Platform at Uber
36mRunning Flink in production: The good, the bad and the in-between
46mKeep it going — How to reliably and efficiently operate Apache Flink
40mGrowing a managed Flink streaming platform from scratch
36mBuilding a Self-Serve Stream Processing Platform with Flink
35mStream Processing for Analysts with Flink and Nussknacker