Building Self Service Platforms for Apache Flink
summary
Panelists compare concrete platforms and tradeoffs: managed offerings (Amazon Kinesis Data Analytics, EMR), DIY on Kubernetes, and internal SDKs that wrap Flink; teams report running over 1,000 jobs on a single cluster, use Flink SQL plus JavaScript UDFs for velocity, and rely on CloudWatch/Elastic/Splunk dashboards, savepoints and automatic restarts for reliability. Expect discussion of multi-tenant resource policies, migration pain (version upgrades), a proposed pipeline-builder API to map high-level pipelines to Flink jobs, and visibility/alerting gaps they still need to fix.
AI-written summary · about
chapters (14)
- 0:00Introduction to panel
- 1:10Panelist introductions
- 12:52SQL and UI solutions
- 16:29User feedback and metrics
- 23:26AWS Flink experience
- 24:52Common user challenges
- 31:51Debugging and education
- 37:11Scaling and customer success
- 38:13Future platform plans
- 44:15Flink community wishes
- 47:49Lessons from early adoption
- 50:36Service level agreements
- 54:54Pipeline architectures
- 57:41Q&A and closing
related talks
36mBuilding a Self-Serve Stream Processing Platform with Flink
50mPanel: Large Scale Streaming in Production
40mGrowing a managed Flink streaming platform from scratch
35mLessons learned on Apache Flink application availability in a hosted Flink service
38mBuilding Unified Streaming Platform at Uber
44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & Beyond
37mBuilding and operating large-scale streaming applications at King
29mThe Platform Engineering Revolution Panel: Empowering Developers Through Self-Service