Big Data Analytics: Immutability & Idempotence

on YouTube ↗11m

" summary

Built a streaming analytics pipeline for games that scales from ~30M to 3 billion events per day, producing ~600–750 GB of JSON daily and using collectors that append to an append-only log in S3 (Kafka-like partitions) while multiple consumers compute metrics. Processing runs in 24‑hour windows with a daily checkpoint and overwrite writes so business-logic fixes are safe and idempotent. Production stores metrics in DynamoDB with >1B entries for a few hundred dollars/month; implemented in Erlang but would now consider Apache Samza/Kafka on YARN.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 1:20Data Flows
3. 3:15Logs
4. 4:15Processing
5. 5:25Idempotence
6. 6:10Checkpoint
7. 9:20Alternatives

" related talks

• 35mStreaming and Real-Time AnalyticsDonnie Prakoso · AWS Summit Online 2020
• 39mLarge scale streaming analyticsJay Palaniappan · Data Con LA 2019
• 38mServerless Data Streaming on AWSAnahit Pogosova · Code Mesh LDN 2020
• 22mTurn Data Into InsightsGabe Hollombe · AWS Summit Online 2020
• 22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million VisitorsFelix Gessert · code.talks 2023
• 53mBig Data: from mammoth to elephantRoman Nikitchenko · JavaDay Lviv 2015
• 35mStreaming and Real-time AnalyticsDonnie Prakoso · AWS Summit Online EMEA 2020
• 19mLog Events @ Twitter: Challenges of Handling Billions of Events per MinuteLohit Vijayarenu · Systems @Scale Fall 2021