Customizing exactly-once processing at scale with Flink
summary
Shows how Flink is extended to get exactly‑once at firehose scale by two concrete builds: a modified StreamingFileSink that enables Parquet row-group commits (uses reflection into the Parquet client, extracts block metadata in Facebook Thrift, extends the S3 recoverable serializer, and writes legit footers on resume) and a lambda‑style two‑phase sink using S3 multipart + distributed locks with resilient retrying commit threads; both run in production and prove durable while decoupling heavy commits from checkpoint coordination.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:50Exactly-once needs in Flink
- 3:37Two-phase commit sink
- 8:22Streaming file sink
- 12:41Bulk format limitations
- 17:23Custom Parquet row groups
- 19:34Transactional sink API
- 22:27Lambda sync for unsupported stores
- 25:43Distributed lock handling
- 27:12Q&A
related talks
26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ Flink
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
34mA journey of scaling high-volume low-latency stateful processing
15mStreaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful Jobs
42mApache Flink deep-dive #bbuzz
43mStream Processing and Applications in the Modern Age
28mIt's Time To Stop Using Lambda Architecture
24mStateful Functions: Flexible like a Microservice, Consistent like a Monolith