Building a Self-Serve Stream Processing Platform with Flink
summary
Intuit builds a self-serve platform that wires Beam and Flink to Argo and Kubernetes, auto-provisions GitHub/Jenkins/artifactory for each processor, and captures lineage by forcing source and sink metadata. Developers get an SDK that emits lag metrics, UIs linking Wavefront/logs/cost dashboards, and namespace isolation for production. Claims include reducing development from months to weeks, 240x improvement in real-time availability, 5x cost reduction, and planned auto-scaling from runtime metrics while persisting state to RocksDB and checkpoints to S3.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:32Overview
- 6:04Architecture
- 11:38Introduction
- 11:53Application Layer
- 15:14User Experience Layer
- 15:42Customer Delight
- 16:48Control Layer
- 18:50Pipeline Layer
- 20:49Runtime Layer
- 23:07Infrastructure Layer
- 25:49Demo
- 32:39Lessons Learned
- 34:07Whats Next
related talks
58mBuilding Self Service Platforms for Apache Flink
37mBuilding and operating large-scale streaming applications at King
36mRunning Flink in production: The good, the bad and the in-between
35mStream Processing for Analysts with Flink and Nussknacker
50mPanel: Large Scale Streaming in Production
44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & Beyond
35mLessons learned on Apache Flink application availability in a hosted Flink service
31mEvolving CI/CD at Flipkart