Scalability Challenges in Big Data Science
summary
Describes a real-time stream-processing pipeline for social network events that scales to millions of users and spikes of millions of events by using NoSQL stores for state, MapReduce for offline model training, and actor-style concurrent workers. Implements in-memory activity counters and histograms with periodic snapshots to serve low-latency queries, and reports that batch MapReduce training is costly, requiring feature reduction and sampling; production use forces trade-offs between memory vs accuracy and streaming approximations vs exact MapReduce results.
AI-written summary · about
chapters (7)
- 0:00Introduction to big data challenges
- 5:35Data processing with MapReduce
- 12:31Real-time data streaming systems
- 19:15In-memory data analysis techniques
- 24:05Scaling solutions for social media
- 27:39Future of big data technologies
- 28:50Q&A and closing remarks
related talks
52m100% Big Data. 0% Hadoop. 0% Java.
24mScalable Stream Processing and Large-Scale Data Analysis
37mBig Data Analytics, Machine Learning, Real Time Processing
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
49mBeyond MapReduce
45mSolving Real & Big (Data) Problems Using Hadoop
19mDemystifying Big Data
19mBig Data