Stream all the things! - life at the end of the firehose
summary
Criteo moves to a hybrid batch/streaming stack: Hadoop for truth, Kafka for transport, Storm for processing and SummingBird to run one job on both online and offline platforms. The system sustains 2–3 million messages/sec (~2–3 GB/s), keeps 20+ petabytes of raw data for replay, and writes ~12 TB/day of logs that get normalized into signals. SummingBird uses monoids to enable parallel aggregation; it works in production but causes Scala dependency and packaging pain.
AI-written summary · about
chapters (21)
- 0:00Intro
- 2:10Stream processing
- 3:38Stream processing characteristics
- 5:28Why use streaming
- 6:56Examples
- 9:19Infrastructure
- 11:20Analytics
- 13:04Data
- 15:13Batch Processing
- 17:10Aggregation
- 19:11Sketch
- 21:50Store
- 24:28No other worlds
- 24:55First job
- 29:18Preproduction
- 34:49Insights
- 38:31Complex Event Processing
- 39:24Single Instance
- 41:52Elastic Search
- 42:34Whats important
- 42:56Log normalization
related talks
42mTaking Hadoop Realtime with Summingbird
01h 01mStream Data Processing for Fun and Profit
34mChasing the Stream Processing Utopia
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
52m100% Big Data. 0% Hadoop. 0% Java.
19mBig Data
37mLet's build our own message streaming platform
57mBig Data Pipelines With Scala - Part 2 of 2