fStream: a managed stream processing platform at Flipkart
summary
Flipkart builds and runs fStream, a closed‑source managed stateful stream platform that handles 500+ jobs at peak 400K events/sec. It plugs Spark or Flink as execution engines, stores state on HDFS, provides a declarative mandatory join model for low‑latency plus eventual correctness, and adds automatic retry/sideline queues for recoverable and irrecoverable records. Developers test locally, use a job repository with versioning and lineage, and get built‑in metrics and alerts.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:25Flash Sale Reports
- 2:47Trending Products
- 4:33Use-case 3: Search Auto-suggest Personalization
- 5:21Sample topology from the above use-cases
- 6:58Why a stream platform?
- 10:03Storm
- 14:45fStream Architecture
- 16:15Programming Model : Source Stream of input data
- 17:27Programming Model : Operators
- 17:59Stateful Operation : Join
- 19:26Concepts: Stateful Join
- 20:25Stateful Operation: Join
- 20:59Stateful Aggregations
- 23:07Testability
- 24:02Fault Tolerance
- 25:47Deployment & Management
- 26:43Monitoring & Alerting
- 28:58Future Work
- 30:53Key Takeaways
related talks
21mFlink's application at Didi
34mBeaming Flink to the Cloud @ Netflix
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
47mBring Flink to Large Scale Production
28mSweet Streams Are Made Of These
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
41mBeyond the Hype: When Apache Flink Solves Real Problems
25mData: Moving with the flow