Getting into Low-Latency Gears with Apache Flink
summary
Practical checklist with measured gains: increasing parallelism cuts 99th‑percentile latency from ~3s to 650ms by reducing per-task CPU load; switching from RocksDB to the in‑memory HashMap state backend can drop state access latency to ~500ms (watch memory and GC); lowering periodic watermark interval and network buffer timeout reaches ~330–370ms; using async I/O for external lookups reduces ~600ms→100ms; incremental window aggregation halves 1.7s→720ms; object reuse and faster serializers cut CPU and bring 4.5s→~0.5s. Flink features used: RocksDB, HashMap backend, checkpoint tuning, buffer debloating (1.14).
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:34Latency & Its Measurement
- 4:03Low-Latency Techniques
- 4:45Direct Latency Optimization
- 21:17Optimizing Throughput for Latency
- 31:12Key Takeaways
related talks
20mTuning Flink Filesystem Plugins for State Persistence
23mBuilding a network stack for optimal throughput / low-latency trade-offs
48mFantastic Performance and Where to Find It
27mArchitecting Sub-Second Analytics Pipelines
29mA brief history of time with Apache Flink
29mWhen your stream doesn't stream very well
36mMake It Fast
46mTaming the 9s