It's Time To Stop Using Lambda Architecture

on YouTube ↗28m

" summary

Argues that Lambda’s dual systems are avoidable by using Apache Kafka compaction or tiered storage for long retention, Kafka transactions plus Flink two‑phase commit for exactly-once delivery, and Flink keyed state with Flink State Processor API to patch or bootstrap state instead of reprocessing years of data; shows practical patterns for CDC, upserts to Pinot/HBase/iceberg, and elastic Kafka clusters for fast historical reprocessing.

AI-written summary · about

" chapters (9)

1. 0:00Intro
2. 0:26Lambda Architecture
3. 2:45Cup Architecture
4. 4:32Why I like streaming
5. 8:40Kafka
6. 14:03Streaming
7. 18:38Data Sync
8. 21:58Use Cases
9. 24:18Audience Questions

" related talks

• 42mApache Flink Worst PracticesKonstantin Knauf · Flink Forward Virtual 2020
• 42mApache Flink Worst PracticesKonstantin Knauf · Flink Forward Europe 2019
• 24mStateful Functions: Flexible like a Microservice, Consistent like a MonolithSeth Wiesman · Flink Forward Global 2021
• 26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ FlinkPiotr Nowojski · Flink Forward Berlin 2017
• 55mDemystifying Stream Processing with Apache KafkaEwen Cheslack-Postava · Philly ETE 2016
• 27mCustomizing exactly-once processing at scale with FlinkAndrew Torson · Flink Forward Virtual 2020
• 36mApache Flink in the Cloud-native EraStephan Ewen · Flink Forward Global 2021
• 38mRe-architecting Apache Kafka for the CloudGwen Shapira · Craft Conference 2021