A Materialization Engine for Data Integration with Flink

on YouTube ↗21m

" summary

A Flink-backed engine reads topic partitions from Nakata (Kafka), deploys one short-lived batch job per request via Flink’s REST API, and writes per-partition files to EBS then S3 while transforming nested event JSON into flat CSV according to JSON→CSV mappings stored in ZooKeeper. Jobs use a stoppable source and expose custom accumulators for progress; a session-window compaction by partition key and ordering key reduces stream size (up to ~70%), leaves out-of-order corrections in separate files, and simplifies operational stack and ingestion into relational warehouses.

AI-written summary · about

" chapters (8)

1. 0:00Intro
2. 2:06Architecture
3. 3:46Data Integration
4. 4:59Materialization Engine
5. 7:13Flink Backend
6. 11:06Stream Compaction
7. 14:37What we gained
8. 16:43Questions

" related talks

• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 31mFlink @WORKDAY: scaling of ingestion data pipelines to S3Leire Fernandez de Retana, Enrico Agnoli · Flink Forward Global Virtual 2020
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016
• 40mFlink in Zalando's World of MicroservicesJavier Lopez, Mihail Vieru · Flink Forward Berlin 2016
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 51mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Voxxed Days Ticino 2025
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016