How Nextdoor improves their customer experience in real time with Apache Flink

on YouTube ↗16m

" summary

Built an Interaction Store that consumes Kinesis Data Streams, processes with Apache Flink on Kinesis Data Analytics, and writes to DynamoDB; state holds HyperLogLog sketches, snapshots run twice a day, and streams retain 24 hours for replay. Production deploy takes under three months. Key fixes: reduce OOMs with operator chaining and operator-level parallelism, cut DynamoDB cost spikes by switching to staggered processing windows, and smooth input with adaptive reads on the Kinesis consumer.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 2:42kinesis Data Analytics
3. 3:33How to build Apache Flink applications
4. 4:23Key benefits of Apache Flink
5. 4:34Introducing Eugene
6. 4:47What is Nextdoor
7. 5:28The problem
8. 8:34Challenges
9. 11:28Processing time windows
10. 12:40Back pressure
11. 15:05Conclusion

" related talks

• 40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache FlinkYing Xu, Kailash HD · Flink Forward Virtual 2020
• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 41mBeyond the Hype: When Apache Flink Solves Real ProblemsNaci Simsek · Berlin Buzzwords 2026
• 45mApache Flink — Real-Time Data ProcessingGaël Renoux · Devoxx France 2024
• 49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing ApplicationsStephan Ewen · Scale By the Bay 2017
• 23mCustomer Journey with streaming data on AWSRahul Pathak · Flink Forward Europe 2019
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016