NRTG: Fast, Systematic & Scalable Flink Application Development at Pinterest
summary
Pinterest describes a Flink framework that compiles a single dataflow class into optimized Flink graphs, adds Kafka sources, async RPCs, and backfill support, and replaces Kryo with a MessagePack-based serializer to handle 80+ types. It reports 3x faster development versus the DataStream API, a 36k-line framework (15k source, 25k tests), and typical signals at ~300 lines of client code with end-to-end CI/CD via Bazel/Jenkins, Spinnaker and Hermes running an 11-stage pipeline with 30+ parameters.
AI-written summary · about
chapters (5)
- 0:00Pinterest Architecture & Business Requirements
- 3:40What is NRTG, Why NRTG
- 9:30The Fast, Systematics and Scalable Promise
- 21:13Onboarding Outcomes
- 22:56CI/CD in the Future
related talks
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
37mBuilding and operating large-scale streaming applications at King
38mFaster and Furiouser: Flink Drift
43mStream Processing and Applications in the Modern Age
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
42mApache Flink deep-dive #bbuzz
19mIntegrating Flink and Kafka in the standard genomics pipeline
44mApache Zeppelin: A friendlier way to Flink