The world wide 60B transaction per day journey

on YouTube ↗36m

" summary

Claim: AudienceScience scales its Helios stack to 1,000,000 transactions per second and handles 80 billion requests per day across six global data centers. The pipeline uses Kafka + Storm + Hadoop with Voldemort for hot key storage, Cassandra for time series, Spark for batch/query, Docker+Mesos/Marathon for deploys, and S3/EBS for cloud swaps. Takeaways: avoid huge single brokers, design for deletes and outliers, prefer dynamic parallelism, and optimize map-side work and output file shapes.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 5:32Dynamic Parallelism
3. 6:41Eventual Consistency
4. 7:28Dynamic Limiting
5. 11:27Nagios
6. 24:59Lessons Learned with the Scaling Cassandra
7. 28:19Clean Up
8. 29:10Move to the Cloud
9. 29:57Object Store versus Hdfs

" related talks

• 36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzzAmeya Kanitkar · Berlin Buzzwords 2015
• 10mImpacting Business as it HappensAnil Gadre · Strata + Hadoop World San Jose 2015
• 19mLog Events @ Twitter: Challenges of Handling Billions of Events per MinuteLohit Vijayarenu · Systems @Scale Fall 2021
• 24mHow To Remove Data Siloes: Tips & ConsiderationsNishant Dixit · Big Data LDN 2022
• 53mBig Data: from mammoth to elephantRoman Nikitchenko · JavaDay Lviv 2015
• 11mBig Data Analytics: Immutability & IdempotenceKnut Nesheim · GOTO Berlin 2014
• 38mServerless Data Streaming on AWSAnahit Pogosova · Code Mesh LDN 2020
• 22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million VisitorsFelix Gessert · code.talks 2023