Growing a managed Flink streaming platform from scratch
summary
Pinterest builds a managed Flink platform that onboards over 25 projects and productionizes 27+ jobs in H1 2020 by shipping a unified source (reads Kafka or S3/mercer files with thrift schemas), a JobService/GMS that automates reliable submissions, failover and de-duplication, and a CI/CD + validation pipeline that runs 12‑hour staging, backfills historical state and asserts results. The platform cuts bootstrap from seven days to 3–4 hours, validates 3× throughput in staging, and recovers across AZs during AWS DNS outages.
AI-written summary · about
chapters (12)
- 0:00Introduction and agenda
- 2:29History of streaming
- 6:56Why Flink was chosen
- 8:20Production use cases
- 10:10Processes and tools
- 13:28Production certification
- 15:49Unified source development
- 23:40Job management services
- 28:22CI/CD and validation
- 32:25Debugging with Dr. Squirrel
- 35:11Flink SQL and future steps
- 37:32Real-world success story
related talks
37mBuilding and operating large-scale streaming applications at King
58mBuilding Self Service Platforms for Apache Flink
50mPanel: Large Scale Streaming in Production
35mLessons learned on Apache Flink application availability in a hosted Flink service
30mImplement Reliable, Isolated & Unified Job Submission
36mRunning Flink in production: The good, the bad and the in-between
12mEngineering Productivity, Infrastructure
38mFrom Flink 1.3 to 1.4